There are many different factors to consider when selecting the type of oversight for a particular simulation project. The relevant factors vary from project to project, and there is no one-size-fits-all answer. This article presents a list of some of the factors to consider, and it is by no means an exhaustive list.
Safety
When considering the level of oversight required for an M&S project, safety considerations are critical. Some products do not present a significant safety risk, so a simulation of them might not consider safety at all. Even so, engineers on these projects should ensure that there are no hidden safety issues.
Some safety issues to consider are:
- What hazardous situations can occur? Mobile systems have collision risks. Electrical systems must not endanger personnel. Hazardous materials must be contained.
- What is the level of harm from a failure? A complete failure of a lawnmower is very different from a complete failure of a passenger airliner.
- What hazardous events can be prevented? A simulation can help design a thermal control system that avoids catastrophic battery failure, or help design a geofencing algorithm to avoid collisions between aircraft and drones.
- What hazardous events can be mitigated? A simulation can help determine weather restrictions on aircraft, or help design algorithms for an autonomous car to avoid other vehicles being driven dangerously.
- What hazardous situations can be controlled? A manufacturer of an excavator can enhance the operator’s situational awareness, but has no control over the behavior of other personnel at the construction site.
The safety issues to consider for an M&S project are very situation dependent, so each M&S team must carefully assess these issues for their particular situation.
Real-World Effect
When considering the optimum types and level of oversight, it is helpful to consider how much the simulation will affect the real world. Different simulations have different levels of effect, ranging from direct effect to no effect, depending on how they are used:
- Design. A simulation used to make engineering design decisions directly affects the real-world design.
- Tuning. A simulation used to tune a system has direct effect on the behavior of the product.
- Robustness Assessment. A simulation might be used to determine the response by a real-world system to an off-nominal situation. Such a simulation will have a real-world effect if the results are folded back into the product design.
- Disaster Investigation. A simulation attempting to re-create the conditions that have led to a disaster won’t have any direct real-world effects, but might influence future procedures and designs.
- Proof-of-Concept or Feasibility Study. A simulation that explores the efficacy of a future product won’t have any immediate real-world effect, but can influence management decisions.
- Exploratory. A purely exploratory simulation for pure engineering research might have no direct real-world effects.
- Marketing. Some simulations drive animations for the marketing department. This has little real-world effect.
The real-world effects of simulations are as varied as the simulations themselves, and the real-world implications of each M&S project should be assessed when determining the appropriate level and types of oversight.
Error Escape
An important consideration in determining the appropriate level of oversight in an MBD project is the likelihood that an undetected error will propagate into an undesired real-world behavior, or “escape”.
This issue breaks down into two questions: likelihood and effect.
The likelihood that an M&S error can escape depends on may factors:
- When launching a space probe, all undetected errors are on board the rocket, so it is imperative that errors be detected prior to launch. (This is a little tongue-in-cheek, but it is also a compliment to the space community for such a high success rate.)
- Physical prototypes of mass-produced products are an opportunity to detect M&S errors before releasing it to the public.
- In a simulation that is only exploring the efficacy of a particular algorithm, as I mentioned in my previous article, there is no chance of error escape during the project.
The effect of an escaped M&S error also varies widely:
- The failure of the guidance and navigation in a high-value satellite is very different from the failure of a cubesat hitching a ride on the booster.
- An error in a simulation of a manufacturing process might result in undesired effects during manufacturing, but without endangering customers.
Of course, no simulation team (or any engineering team) should rely on downstream checks to trap their errors, but it is a consideration when determining the appropriate types of oversight.
Industry Standards
Many simulations will be in support of, or guided by, standards and regulations specific to their respective industries. None of the widely-used standards actually specify the types of oversight required for an M&S project, because it is so dependent on the circumstances of each project. Some standards specify particular types of oversight, and require an individual project to justify any deviations from that standard. Other standards are more goals-related, and leave much more leeway for the practitioner to achieve those goals.
Most industry standards are managed by one or more organizations. Some of these organizations are specific to particular industries, such as SAE or IEEE, while other organizations are broad-based standards organizations, such as ISO or ANSI.
When determining the appropriate oversight for an M&S project, guidelines developed by experts in your industry are a great place to start.
Legal Requirements
Some types of oversight might be legally required, depending on your industry and jurisdiction, and might also depend on the jurisdiction where your product is deployed. When determining the types and level of oversight for an M&S project, legal compliance is a very high priority, second only to safety considerations.
Most legal requirements that impact engineering involve environmental and safety issues, although many other types of legal requirements are in force. Discussing all of the legal requirements for all industries in all countries is beyond the scope of any single article, but you must be cognizant of those that apply to your project.
In my next article, I’ll discuss some of the trade-offs to consider when choosing the appropriate levels of oversight for an M&S project.