This is a follow-up to my previous post, which describes what I mean by “Distributed Model-Based Design”.
In my career, I have worked on several distributed Model-Based Design projects, and noticed problems common to all of them. This post discusses my insights.
The problems were always encountered during the integration phase, when the contributing organizations would submit their subsystem models and the sponsoring organization would try to integrate them into a whole. The integration effort was always a lot of negotiation with the contributors, and we often had to modify the submitted models so that they could be integrated. This was always a time-consuming effort that was stressful to the team.
Many different problems were encountered:
- Some models were not generating the requested outputs so that they could not drive the inputs to other subsystems. There were two solutions to this: The integrating team could modify the non-compliant subsystem model to get the needed outputs, or the integrating team could go through an often-complicated management chain to request the needed changes.
- Some models needed inputs that had not been considered when initially designing the simulations, and it wasn’t always clear how to supply it. For the integrating team, the first step to resolving this was some detective work to determine from where the signals should be coming. Typically, these were from some other subsystem model, and there would be re-negotiation with at least one other subsystem team to get the needed signal.
- Each subsystem model included a start-up script to initialize it. Coordinating these scripts into a form that could drive the integrated simulation was a nightmare. It would invariably involve building a master script, and then modifying the submitted scripts so that they could be called by it.
- System-wide parameters, such as air pressure or gravity parameters, were difficult to coordinate between the subsystem models. This involved a search to find where the system parameters are referenced in the scripts and in the models, then updating either the references or the referenced variable to use the correct value.
- Monte Carlo simulations were especially challenging. Setting up for a Monte Carlo simulation often involved significant changes to the subsystem models and scripts so that the appropriate parameters could be automatically updated.
- It was not always clear what outputs were needed from the simulation to properly analyze the overall system, and it was not always clear which subsystem should generate the outputs. Addressing this sometime meant that the integration team would need to collaborate with the simulation architects to determine what was needed, and then modify subsystem models to pull out the appropriate signals.
Often these problems would arise when dealing with legacy models. As described in my previous post, sometimes a group will be commissioned to provide a subsystem model because they have an existing model already. Inevitably, the legacy model will require modifications to update it to the needs of the new simulation. This typically requires careful interaction between the groups to ensure the needed changes are implemented.
In extreme situations, I have seen legacy models delivered as-is, without modification. Usually these situations were handled by negotiating a new understanding of what was needed. In one case, it was necessary for the integrating team to spend time getting up to speed with the architecture of, and engineering in, the subsystem model in order to incorporate it into the simulation.
As the above descriptions show, addressing the problems of distributed Model-Based Design often involved the integrating team modifying the work of other engineers. Although necessary, it often undermined engineers’ sense of ownership which was not good for team morale.
In addition, the changes by the integration team caused a different problem when diagnosing bugs: Is the error in the subsystem model or in the modifications made by the integrating team? Diagnosing errors could involve considerable back-and-forth and would sometimes require management intervention.
The above list of problems does not capture the magnitude of the difficulties. Most simulations start small, sometimes as a low-fidelity proof-of-concept, and grow into high-fidelity simulations. This is always done iteratively with many small updates to the simulation. Subsystems are added, removed, and split. Parameters are added, made more complicated, and re-defined. Each iteration of the simulation would be a new integration effort, and some or all of the solutions described above would have to be re-implemented, sometimes from scratch.
On these projects, we used traditional software versioning tools in an attempt to manage the churn. Unfortunately, these became little more than glorified code repositories because they were difficult to coordinate across multiple groups, and updates required a new integration of the entire simulation. We needed a way to update only the affected models without updating the entire simulation.
Because of the on-going integration effort with each iteration, every distributed Model-Based Design project I worked on required at least one full-time engineer dedicated to integration through the life of the project. I encountered extreme cases of this as a consultant on two projects that required an integration team that grew to four full-time consultants over several years.
Recruiting and retaining engineers to work full time on to focus on the integration is difficult, because this not exactly a coveted position on the team. It is not a creative position, and it would require deft handling of the interdepartmental issues encountered.
The problems described above would always spill over into the rest of the engineering project, in the form of delays. For each iteration, the updates were requested by one or more organizations in the overall project, and those organizations would have to wait as the integration team worked through the issues described above.
In summary, the problems encountered in distributed Model-Based Design result in increased head count, higher frustration on the team, and delays in the project.
In my next blog post, I will describe SystemBlend’s unique solution to all of these issues in a simple, streamlined fashion.