Developer Nation Blog

I ‘m part of a hardware research group at Telefónica Digital called “Physical Internet Lab”. Three years ago we started a small group under the Emerging Technologies area of the company focusing on the Internet of Things. The commitment of the group was (and is), in ambitious terms, “to democratize the Internet of Things” opening it to as many makers, developers and users as possible. Our goal has been not entirely altruistic: Telefónica as a network operator has a lot of value to add in the Internet of Things economy.

On day to day basis we build prototypes and products, usually connected objects or components like the Thinking Things building blocks.

Setting up the lab three years ago was no easy task. We wanted to work at the crossroads of the Internet, the Things and the People. But our development skills were almost 100% software related. In the process we built a team skilled on all three sides. And we figured out how to do agile hardware.

Of Agility and Hardware

We ‘ve come full circle. Telefónica I+D (the Telefonica Digital development branch) was created 25 years ago to produce hardware innovations such as X.25 and ATM switches. We did that in the classical engineering fashion: writing long and rigid lists of requirements, splitting the work across solution providers, integrating and then testing following a waterfall schema.

Over time Telefónica I+D adapted quickly to the technology changes and by the mid-nineties we were developing mostly software. First we followed the same engineering process; then we moved towards more iterative methods. In the last 10 years we have adapted fully to agile methodologies.

As we were building the laboratory we found ourselves getting back to hardware. But the company now could not understand a slow-moving unit. The lab had to be agile. So we had to bring agile methodologies to hardware development.

The first difficulties came with the corporate facilities. Hardware work demands physical proximity and we could not afford to have a distributed team depending on collaboration tools on the Internet. At the same time, soldering fumes or drilling noises were not welcome in our modern, bright, open spaces. So the team had to move to a closed office in an old building in Madrid city center.

Moving to the city center was a boon: in minutes we could reach many shops and services, buying anything from hammers to plastic boxes. Visitors now found it easier to visit us in a centric garage-like office. This was great for our open approach as we wanted to help and interact with other companies and organizations.

Purchasing tools was another problem. The corporate procedures were tuned for large-scale purchases such as server farms or external services. Buying a handful of resistors for 10 euros could take several weeks, creating bottlenecks to our work. Fortunately the purchasing department showed a great deal of sensibility. We worked together to redesign the process. Now we buy any component or tool in a single day while still working by the book.

Putting together the Agile team

Hardware work implies multiple teams across several companies with extremely specialized profiles. When setting up the lab we opted for a small and autonomous team, able to build a hardware prototype with no external dependencies.

A small team allows us to work closely integrated, in the same location, continuously coordinating our work. A small team also means that budgets are smaller and is well suited to experimenting, failing, learning and adapting.

Basic agile methodologies such as Scrum expect some degree of overlap between the specializations of team members, so that different people can execute the same tasks naturally balancing the work load. But hardware work is different. It demands a lot of specialization. In our case most of the tasks can be executed only by one team member. As a result, the Scrum methods and tools have to be modified to reflect this reality.

Our internal workflow follows many steps. The first step is the Industrial Designer, a role which is somewhat of a novelty in the Telefonica Digital payroll. Carlos (that’s his name) starts his work in the CAD station designing the physical product: plastic pieces, metal straps, cloth, magnets. Then he builds the design using the currently available 3D prototyping tools such as the laser cutter, the CNC tool (i.e. a computer controlled drill) and a variety of 3D printers. These tools give much flavor to the lab.

In some cases we start from an existing object that we hack so that we can explain a new concept. Carlos at the same time designs and builds, which is a bit out of his job profile. Software developers are multi-taskers, too – they design and type, while software architects can also code. In the hardware industry this is somewhat unusual and typical engineers expect someone else to physically build what they have created. In the lab we follow the software philosophy. It is leaner, and gives the designer a real feel of the piece or circuit construction. This approach demands some tolerance and patience from engineers who have to get their hands dirty.

The same philosophy applies to the next step in the workflow: the electronics engineering part. The electronics engineer first designs new circuits, then prototypes them. We even design and build the PCBs to check that everything fits in place.

The agile doctrine underlines the importance of early user testing. Early use provides rapid feedback focusing the most important characteristics of the product and showing what isn’t relevant for customers. To shorten the time-to-test we use 3D printing and prototyping technologies.

In electronics engineering we massively use Open Hardware. Open Hardware gives us access to lots of ready-to-use designs that we can employ in product testing. In a sense, Open Hardware behaves now like Linux and Open Software in the mid-nineties. It allows us to focus on the real technical or design challenge rather than reinventing the wheel for every test.

Electronics and physical design teams work side by side, so they can verify in real time how components fit in the same object. Our objects become more than simple plastic boxes, as they are tightly coupled with the internal electronics.

Electronics engineers work also with the firmware developers. The firmware developers write the code for the embedded microprocessors. They also have to deal with connectivity issues and power management.

In our Physical Internet Lab, electronics and firmware engineers work side by side. In most situations knowing what will firmware do simplifies hardware design. Similarly, software developers can ask for fine changes in the hardware designs nearly in real time.

On the other side of firmware sits backend development. In our typical systems architecture, distributed devices communicate with a backend service in the cloud. We push as much intelligence as possible to the backend service, so our designs can evolve without touching the deployed hardware or executing firmware updates. We like to think that the back-end gives every object nearly infinite computing power and knowledge, as it can interact with any other Internet service.

Again back-end and firmware developers work side by side. This tight collaboration resolves any integration problems before they appear, and encourages electronics and firmware developers to take issues to the more powerful (and more agile) back-end platforms.

The final technical step is the front-end development, usually based on web and native apps. Again we do a lot of work locally in the lab, well integrated across the team.
The frontend is also tested in complete end-to-end scenarios. Automatic testing tools execute scripts that run against the firmware and the frontend.

And of course, there is a Quality Assurance side. We are extending continuous integration, test driven development and automatic testing to the embedded firmware. At the same time we have to handle more hardware specific tasks such as sensor calibration, assuring robustness and strength.

Physical Interaction Design

The web/application interface and physical design are the two endpoints of the “development chain” of our group. They form the two interfaces exposed to the final user. At the final part of our workflow, the physical interaction designer, works with both web / app and physical design.

The physical interaction designer is responsible for the design of the connected object as a whole. He takes care of building a single object with a coherent interaction model in the physical world and in the Internet.

Without the physical interaction designer we would have to separately design the physical object and the application or web interface. The result would be a split-personality product, usually an amalgamation of data stuck on top of a square box. The physical interaction designer combines the capabilities of the physical object and the Internet interface in a coherent manner.

Physical interaction design, bringing together the Internet and physical objects is a completely new field. There are a handful of specialized schools in the world, and we are working too with UX designers with strong industrial design background.

Everyday physical objects have usually long stories and designs optimized through centuries of use. We still have a lot to learn on how to take the Internet beyond of the smartphone/tablet/PC onto this physical object world. Customers will not adopt Internet of Things devices if they are a step behind of the design standards they have become accustomed in software interfaces.

Agility plays a role here, once again. Developing and prototyping quickly we can try interaction designs with users, test our assumptions and build a sizeable bunch of knowledge around user interaction with connected objects.

External providers

Of course we have to work with external providers, especially when dealing with complex technologies or industrialization. For development we often use online services for as PCB manufacturing or 3D printing. They are extremely easy to use, robust, fast, and offer a direct web interface instead of long negotiations with a salesperson.

For the final manufacturing we interact with real, serious manufacturers. Agile, as a software development doctrine has no solutions to this task. But Agile can be seen as a spin-off of Lean philosophy, which was created to deal specifically with manufacturing issues.

One of the main lessons from the Lean methods is that service providers have to be tightly integrated in the business process. We have found this is very important also for us. The lab has spent considerable efforts building trust relationships with service providers and manufacturers, integrating their teams with the lab. Schedules and plans are shared under an openness philosophy. We have established even real time communication so their teams get continuous feedback from the engineers in the lab.

The future of agile hardware

We have yet a long way to create a truly Agile Hardware lab. Physical work is sometimes slower than software development. Some other times (especially when prototyping on Open Hardware designs) they are blindingly fast and have to pause and wait for software components. Speed differences keep the group working on different “user stories” at the same time.

External dependences are many, and the lab will never be, in that sense, completely autonomous. But we can find yet faster service providers and build leaner and more integrated workflows with them.

Regarding Quality Assurance we have to handle correctly the physical device characterization and fit the expensive and slow certifications in the product workflow.
The bright side is that Agile methodologies provide and require continuous improvement. Every sprint or work cycle forces us to learn and adapt our methodology and organization, looking for a better process. Perhaps in a couple of years we’ll have a completely different process in a completely different lab, and it will be all right.