\ ^__^
\ (oo)________
(__) )\/\
||----w |
|| ||
This is essentially a story time about a project I worked on with a small team of engineers in the automotive industry. I'm proud of it because it was one of the first times I was allowed to take the technical lead on a project, and because it was ultimately a success story. We had our ups and downs, technical and inter-personal challenges. But, we delivered a completed product on schedule. And, the client was happy. Names and places have been changed to protect the innocent.
Our story starts, as all good fairytales do; in the German Countryside. A company based there had developed a gizmo for use in garages to connect to the computers onboard cars and provide diagnostics to a technician. We'll call them AutoComp. The gizmo was a small handheld thing, with a large connector on top, a screen, and a few buttons. We called it a Tricorder. It looked like a Tricorder from star trek, and while Tricorders helped technicians on the show debug and monitor all manor of scifi nonsense, this thing allowed the same sort of thing for cars. So Tricorder fits!
The Tricorder could communicate with the car over a bevy of protocols, including serial, CAN and Ethernet. We were bring brought in to enable Diagnostics over IP (DoIP).
So me and my team went to Germany. We had me, two other engineers and a project manager. We had 3 days on-site with the team at AutoComp. We had to learn everything we could about the project; the client's requirements, how to setup a development environment and build their firmware, learn about DoIP and collect the specification. We found their development environment. They were using an older version of Qt and would build the firmware using a version of Ubuntu which was already in LTS. To speed along development, I setup their tooling in a container which allowed their firmware to build in an environment it expected without having to change much on my work computer. AutoComp told us they wanted us to target C++14 and use modern patterns.
We also got to see a prototype implementation. Prior to our involvement, AutoComp had commissioned a prototype to see if the project was at all viable. We got the sense that they had hoped the engineers who made the prototype might have been able to develop the prototype into the final product, but the architectural problems in the prototype were obvious. It wouldn't be able to scale well because of it's liberal use of global variables. It was written in one file and it was a spaghetti code. AutoComp told us that we could look at the prototype for reference, but we were not to use any of it's code. We also had access to a third party simulator application which would essentially pretend to be a car.
After that, we went back to England, We would have 1 week to come up with a plan. If AutoComp didn't like the plan, or thought that we weren't capable of meeting their expectations, they would have the opportunity to axe the project right there and then. The team was a little apprehensive, given that the team who created the prototype had already been shown the door. But I wasn't worried. I had seen their codebase and I had a copy of the DoIP spec for light reading on the plane. The module we had to create was already taking shape in my mind's eye:
We would implement a message pump system, wrapping the DoIP socket in an IOStream object to take advantage of STDlib's type conversion mechanism. We would dispatch the data to different state machines to handle specific DoIP sequences. The simplest of which would be basic lambdas. State Machines would allow the complicated sequences to be handled logically and asynchronously.
We were asked to use C++ features, so the state machines were to be allocated on the heap with "new" instead of doing something interesting with the stack (as would be my embedded engineering instinct). All in all, I thought the project would take about 3 months to deliver.
I ran the plan by my team. They turned their nose up at the state machines, but they were reassured that the majority of the sequences in the spec could be handled without having to be re-entrant, simplifying matters substantially. With the team onboard, the plan was presented to AutoComp. They approved and the work could go ahead.
We divvied up the work and we started in earnest. One of my colleagues would work on the socket side. Reading the raw data and constructing packets. The other was responsible for interfacing with AutoComp's application. I was responsible for the overall application structure. With those bits in place, we would all have the responsibility of implementing DoIP sequences and reviewing each other's code and writing unit tests.
Development proceeded quickly. It wasn't long before our module was able to interface to the simulator. My team was small, but highly competent. The work proceeded mostly without incident.
There was one major point of contention on the team. One of my colleagues had implemented a sequence as a lambda and made the assumption that all the packets for the sequence would arrive … sequentially. And with no packets for other sequences before it's sequence was completed. It was a fair assumption to make. Because the spec didn't explicitly say that packets for other sequences were allowed in the interim, it also didn't say they were forbidden. It simply implied time could pass. My argument was that even if the sequence passed 100% of the time on the simulator, because the spec was ambiguous, other car manufacturer's might not be so predictable. Ultimately, the implementation proved unstable on the simulator, and the fix was straight forward and known; reimplement it as a re-entrant state machine.
The work proceeded to plan and was on schedule. We were able to test the application by both running it on a computer and also running it on a Tricorder. Because it was over IP, it was trivial to connect it to the simulator over a common network connection. We implemented the whole spec, but before we could deliver the solution to the client, we'd need to try it on an actual car. Luckily, someone in our company had an AutoComp car that was supported by the hardware. He was the owner of the company. He let us try our application on his car!
That experience was a little unnerving. Realistically, there's not much chance that our application or the client's device would do something to break his car, but if it did, it wouldn't exactly be covered under warranty. We plugged the Tricorder into my boss' car and loaded our application and in the moment of truth… nothing happened. We investigated further and we couldn't find any issues with the module we had written. In all cases it would work perfectly on the simulator but it was as though the Tricorder couldn't see the car via our module.
It was time to deliver the module to the client, we were worried about it being non-functional. But we delivered it non-the-less. AutoComp looked into it and found that the large connector which interfaces with a car was actually wired wrong. Ethernet was fundamentally broken on the device and it would never work on that device. We got paid for the work, but it would take a year before AutoComp would release a new version of the device without the hardware bug. Eventually though, on a V2 Tricorder. Our application worked flawlessly and without modification!