Some simulation projects require minimal technical oversight. I had the pleasure of working with a brilliant colleague who was exploring a novel terminal guidance algorithm, and he built a simulation with the goal of understanding the efficacy of the new algorithm. To accomplish this, the simulation implemented the algorithm, and coupled it with a notional missile model that was merely enough to exercise the algorithm. Because there was no real-world system involved, and because the simulation was not directly influencing any engineering decisions, there was minimal need for engineering oversight. (Obviously, if this algorithm were ever applied to a real-world system, the appropriate oversight would be applied at that time.)
At the other end of the spectrum, some simulation projects require extensive planning, controls, and documentation at every step. These are projects that guide the design of safety-critical, high-value, or high-profile real-world systems. Examples of this are well known, such as self-driving cars, passenger transport aircraft, and spaceflight systems.
The majority of simulations fall somewhere in between these two extremes, and the appropriate types and level of oversight depends on each particular situation.
Definition
The difference between project oversight and project management is not always clear, and often overlap, so I’ll define technical oversight as "input from outside the simulation team to control simulation correctness". This is intended to include activities that are required by outside personnel but carried out within the simulation team.
The goal of this definition is to distinguish technical oversight from:
- Decisions made internally by the simulation team, and
- Non-technical influences from outside the team, such as budget and schedule.
Types of Oversight
The types of oversight that might be applied to a simulation project are as varied and as numerous as the project managers that are leading the projects, so I will not attempt to list them all. Instead, I want to discuss a few to convey some idea of the variety.
Technical Reviews
A common form of oversight on simulations is technical reviews. The goal of a technical review is to gather input from stakeholders. This includes:
- Peer Reviews. This involves having the simulation team review the work of an individual on the team. This is an example of an oversight task that might be required by outside personnel but is carried out by the simulation team.
- Design Reviews. This involves review by stakeholders outside the simulation team.
- Customer reviews. This is a design review specifically for the customer. For one-off products, such as a large passenger ship or a spaceship, the customer is easily identified and is directly involved in the review. For mass-produced products, the marketing department might stand in for the customer.
- In-Person Meetings. Despite all of the collaboration tools that have been built, sometimes there is just no replacement for in-person discussions. COVID 19 taught us how to stumble through on-line meetings, and it also taught us that sometimes those on-line meetings don’t work.
- Source Review. This can be a meeting in which the developer takes the audience through the code line-by-line (or block-by-block). Alternatively, it might involve making a release copy of the source code available to reviewers to inspect on their own. Most engineers do not like this level of scrutiny.
- Document Reviews. There are two goals to document reviews. First, a document review is intended to determine whether the document itself adequately covers the topic. Second, a document review is intended to determine whether conclusions presented in, or implied by, the document are sufficient. A document review might replace a source review.
- Frequency and Scheduling of Reviews. Are there regularly scheduled reviews, or are they tied to milestones?
- Review Team. Deciding who should review the work can make a difference. For military projects, this might be constrained by security considerations.
Change Control
In the context of this article, "change control" does not include the CM tools that any software team would typically employ. Instead, this implies the processes put in place to help manage correctness.
- What elements should be baselined? A simulation team might baseline the requirements, the design, and/or or the simulation itself.
- Gating Baseline. What baselines must be in place before the team can proceed with the next stage of development? It might make sense to baseline the requirements before initiating the design effort. Similarly, it might make sense to baseline the design before beginning implementation.
- Documentation of Changes. What variations from the baseline should be documented?
- Change Process. What is the procedure for changing a baselined element?
Requirements
The requirements imposed on a project can range from implied requirements with no controls to very formal requirements with documentation, reviews, and baselining.
- Written Requirements. Simulations of high-risk systems probably should have formal requirements. Exploratory simulations that support proposal development might not have any written requirements.
- Requirements Development. Does the project include a formal requirements development effort?
- Requirements Review. Who should have input into the requirements?
- Requirements Validation. What processes are in place to ensure the requirements make sense?
- Requirements Compliance. What processes are in place to ensure the simulation complies with the requirements.
Documentation
Documentation in support of technical oversight is usually not a stand-alone effort. Instead, the documentation is in support of other aspects of technical oversight.
- What Gets Documented? Usually, any baselined element is accompanied by thorough documentation. Written requirements form a document. What design and implementation decisions should be documented? There are many opportunities for documentation, and you should judiciously select what documentation to generate.
- When are Documents Produced? When employed, documents are typically produced on certain events, such as whenever a formal review is conducted, a baseline is established, or a change is requested. There might also be regularly scheduled documentation, such as progress reports.
Not all documentation is directly related to technical oversight. For example, non-technical administrative documents and user’s guides are needed, but might not contribute to technical oversight.
Calibration and Tuning
Some simulations, such as a simulation to test the control algorithms of an autonomous vehicle, require some combination of calibration and tuning. Other simulations, such as a proof-of-concept simulation, might not need calibration.
- Calibration Procedures. What procedure should be used to calibrate the simulation? Are the calibration procedures specified in the technical oversight, or are they developed by the simulation team?
- Source of Calibration Data. What level of control over the calibration data is required? How thoroughly should the pedigree of the calibration data be controlled and documented?
- Tuning Techniques. Are the system tuning techniques specified in the technical oversight?
Standards and Regulatory Compliance
Not all oversight comes from your project. It can include requirements developed by third parties outside of the project.
- Regulatory Requirements. You must comply with all legally imposed requirements. What processes will ensure compliance with all regulatory requirements? What processes will ensure that all applicable requirements are addressed?
- Industry Standards. Most industries publish standards for best practices, so these might be a good starting point. Some industry standards have quasi-legal implications, and these must be addressed appropriately.
- Technical Standards. A simulation might be required to comply with certain technical standards to help ensure interoperability.
- Company Standards. Your organization might have internally-developed standards that are applicable to simulation projects.
Lifecycle
Project lifecycle doesn’t necessarily have technical impact, but it is sometimes required as a mechanism to manage technical correctness.
- Lifecycle Model. Most tightly controlled projects will employ a V-Model or a waterfall model. Agile organizations might prefer spiral development.
- What events are specified in lifecycle? If the simulation is always run by the development team, then the lifecycle might not include a formal release. Spiral development might not have a formal, baselined design.
- Milestones. Depending on the lifecycle model, the transition between phases of development might be define by certain milestones. Examples include a requirements baseline, a design release, or releasing the simulation to the end users.
Again, this is by no means a complete list of the types of technical oversight that can be employed on a simulation project, but hopefully this will help you think about the types of oversight that are relevant to your particular project. Some projects will employ most of these techniques, while other projects will incorporate none of them, but the majority of projects fall in between these two extremes.
In my next article, I’ll discuss some of the factors to consider when deciding the type of oversight that is appropriate for your project.