This article is the second part of a two-part series discussing how to develop mission operations (MissionOps) for space.
In this piece, produced in collaboration with MissionOps software developer Epsilon3, a paying participant in the satsearch membership program, we take a closer look at what a good mission operations framework and system requires.
You can read the first part of the guide here, discussing what MissionOps is, providing examples showing why it is so important, and sharing the common challenges that are faced when a new mission is designed.
Creating high-quality procedure documentation
Mission procedures and protocols need to be clearly written down beforehand in order for engineers and operators from different teams to have the information they need to do their job. This is particularly important in the case of errors or emergencies.
All documentation should be clear, with as little room for ambiguity or confusion as possible, and it should be kept up to date. The updating process itself should be carefully tracked and be verifiable by all relevant users.
Documents also need to be clearly organized and accessible for all of those that may need them, including personnel who could be added to the team later. The procedure for granting access to relevant documents needs to be well-established, simple, and managed by more than one person (in case the original administrator is unavailable when a new hire begins work for example).
It helps to have a named owner assigned for each document, and at least one back-up owner in case they are unavailable. The document owner should be responsible for keeping it up to date, ensuring it is accurate and accessible, and backing it up on the assigned schedule.
All relevant team members should also be able to easily find out the owner of each document at any time in case of questions or comments. Good documentation practices are based on logical and sensible processes. Think carefully about who needs access to what information, at what time, and in what format.
In the early stages of a mission it may seem excessive to spend significant time developing strict processes on document management, and ensuring that existing materials are properly organized and set up for effective use. But any weaknesses in the system will compound and this work has the potential to save far more time, effort, and money in the case of an avoidable failure further down the line.
However, good documents are only part of the picture. The tools used in how those documents are used and discussed, along with other mission assets, is also vital.
MissionOps software and electronic procedures
Whether developing a new flatsat or operating a rover on Mars, every team requires a framework for managing MissionOps that meets the demands of modern space missions and the expectations of people used to interacting with fast, intuitive, and high-quality software on a daily basis.
This is not just about the command and telemetry system, but about the whole tech stack involved, including command and control (C2), data storage, GUIs, procedures and timelines.. All of the different frameworks, tools, libraries, software, and programming languages required need to work effectively together in an efficient mission operations system that adds value to every project.
The electronic procedure software solution developed by Epsilon3 is designed to meet this need.
Epsilon3 is a US-based provider of mission and engineering operations software. The team consists of engineering and design professionals from firms such as Northrop Grumman, Google, and SpaceX, with experience that includes first-hand operational management of sending American astronauts to the ISS.
The Epsilon3 platform digitizes, integrates, automates, and manages different kinds of operations in a single digital environment, built with the specific nuances of space missions in mind.
The Epsilon3 team is aiming to develop the industry standard of operational software with their web-based procedure tracking and communications platform. It is designed to support the entire project lifecycle by creating, running, executing, and track all procedures for:
- Integration and testing,
- Ground control training and flight operations,
- Human spaceflight training and operations, and
- Satellite/constellation integration, testing, and operation.
The Epsilon3 system is designed to enable more seamless communication between operators, engineers, technicians, and the test bed or a spacecraft in orbit for command and telemetry purposes.
All of the plots and reports generated are available to everyone accessing the system. There are no artificial barriers or information silos erected, but ownership of different aspects of processes and procedures can be assigned and transferred with ease.
The software also integrates with tools such as Jira to enable insights and lessons to be extracted from issues. This closes the loop on a specific issue automatically, and is one of the many advantages that a purpose-built digital environment can bring to space missions.
The value of adopting electronic procedure software
There are a number of advantages that utilizing an integrated procedure management software platform can bring to a mission, such as:
- Simpler management of complex procedures,
- The reduction of errors due to human operator use,
- Embedded release processes for relevant materials and resources,
- Integrated command and telemetry capabilities to natively review data,
- Real-time synchronization of the web-based platform across users (no checking if “this is the latest version?”)
- Synchronization of communications to create update logs and histories that can be checked as needed,
- Continuous improvement of operations based on user activity,
- Ensuring all engineers, managers, customers, and other stakeholders are on the same page, with established procedures, common terminology, and transparent timelines, and
- Improvements to culture, cohesiveness, and inclusiveness that come with enhanced collaboration and communication.
Using the right tools for mission control operations not only saves time, effort, and money, but also enables users to collect more data and create more value. If a space asset is out of commission for any length of time there is a potentially large cost to the owners to fix an issue, along with lost revenue for data that can’t be collected.
Efficient MissionOps are about de-risking missions and doing everything possible to ensure success. In space there are so many environmental variables that we can’t control, but we can be prepared for them, understand their impacts, and build plans to mitigate their effects.
Training operators on mission operations
Regardless of the documentation approaches and MissionOps platform in use, it is important to comprehensively train all operators and back-up personnel on the processes and tasks relevant to their work.
Each mission and organization will have established training approaches and programs, but here are a few pieces of advice based on Epsilon3’s experience in a wide variety of programs, from high-profile crewed missions to smaller technology demonstrations:
‘Train like you fly’ – do complete walkthroughs and then runthroughs of missions – make sure that everyone participating has the exact setup, software, data, decision trees, and protocols for the mission. They need to know exactly how to record and track what they are doing, and multiple people need to know how to access this information.
Such sessions need to mirror the live operations as closely as possible. Think about how you can replicate the exact environment, timings, systems, and other elements of the actual flight.
Consider all contingency plans – write down all possible things that could go wrong and come up with a plan B and C for each of them, both individually and in compound (e.g. if several things went wrong at the same time). Select and test out those contingencies most likely to be used first, and most often.
Train in scenario-based responses – train for specific instances with things going right and things going wrong, both in space (e.g. hit by debris, lose contact etc.) and on the ground (e.g. an operator gets ill, you get hacked, there’s a power outage at the ground station etc.)
Develop vocabulary around such scenarios so they can be more easily understood, discussed, and recalled when needed.
Test decision-making in high-stress situations – make sure that people can act effectively under stress and that they use the right resources for support when needed. Also assess whether the systems, software, protocols etc. hold up under stress. Fix any roadblocks and increase resources if needed.
Repeat such tests, at varying levels of intensity, and by switching people around. Ensure you debrief and learn from each experience to improve operations where needed.
Additional guidance for mission teams
In this section are a variety of other useful guidelines for operational teams, based on the experiences of Epsilon3 personnel, that will help you build a better MissionOps system:
Prelaunch
- It is best to be in touch with stakeholders, such as licensing regulators, ground station providers, logistics companies, and all third-party suppliers (at least those you plan to work with in the current schedule), at least a year in advance of the mission launch date, if not earlier.
- However, before initiating such contacts, ensure you have a working concept of operations (ConOps) plan – and then you can gather the assets you need to apply for licenses and engage with external service providers. But this is iterative; their needs and requirements will change the ConOps plan as you go along, and you need to be able to adapt without too much administrative overhead.
- Ensure that you hold difficult conversations before launch – to share information, establish the ownership of the various decision-making processes, and to practice communicating under stress.
- Determine how often you can communicate with the spacecraft, how much delay there will be and your ground station’s availability. You need to have this information clearly recorded and readily available to anyone that needs it.
- You need to decide/determine whether and how the data is encrypted – on the satellite, in satellite-satellite communications, and/or in satellite-ground communications. You need to understand how the data will be held, in real-time, and processed. This involves mapping out exactly where data will be stored and how easy it will be to access, as well as what post-processing will be performed, either automatically or manually.
During the mission
- Structured reporting on mission milestones and tasks is very important – it saves time and communicates more information more easily. Just ensure that all relevant personnel understand the structure and nomenclature of reports, as well as how to create and review them.
- Comprehensive communications and operations plans and protocols should be established, covering details such as:
- Who sends commands, and who supports, at each stage?
- What is the call-in structure and system? Who needs to be on-call and when?
- What commanding decisions can and will be made by the ops teams, and when will they wait for potential specialists?
- You need to assume a certain level of knowledge of the recipients when writing reports or communicating on issues in order to be efficient (otherwise you would have to explain too many things under time pressure). But don’t assume the people reading reports will understand everything intuitively and implicitly. If in doubt, just add a few extra details.
- People will always approach issues in terms of their background knowledge and area of expertise, but you need to communicate in a way that anyone in the team can follow. Again, standardization and structure is your friend.
- In many cases you waste time later by not asking questions earlier. Get everyone on the same page and then you will be able to make more progress faster.
Protocols encoded in mission documentation need to cover all aspects of the mission output, which will include a complete understanding of data privacy, security, and encryption.
Ensuring security in space missions
Security is a critical issue for space mission operations. Integrated operational software collects all mission critical information in one place and so the security standards need to be at the highest level possible.
Epsilon3, for example, ensures security in a number of different ways. The software has 256-bit AES encryption at rest and SSL/TLS encryption in-transit. It also protects against data loss with recurring encrypted backups.
Aside from encryption, the system also offers flexible deployment through the Amazon Web Services (AWS) GovCloud for International Traffic in Arms Regulations (ITAR) compliant data storage or with your own on-premises servers.
Roles-based access control can be set up for restricting permissions to data across the organization and multi-factor authentication is also available to secure access.
Epsilon3 also periodically undergoes System and Organization Controls (SOC) examinations to ensure its systems have effective control over the security, availability, and confidentiality of the online platforms. An SOC type 2 report is available upon request.
Security issues in MissionOps needed to be approached in the same manner as other engineering and operational considerations. Clear, simple, and well-established procedures need to be in place under the responsibility of relevant personnel, and all of the mission team should be aware (or should be able to easily find out) what the protocols are.
Conclusion
A space mission is complex enough without adding massive overheads when managing communications, documents, and people. A good MissionOps plan will reduce such overhead and make all aspects of the operation more efficient and seamless for the user.
All of your software systems and teams need to talk to one another in a consistent and traceable fashion, with common tools and approaches that remove the need for low-level decision-making.
Whether in low-pressure day-to-day activities or in critical emergencies, operators shouldn’t have to answer questions like
- Where is the document I need?
- Who should I share this error report with?
- How do I find out when we have ground station time?
- Who’s in charge of this part of the mission?
These questions should be answered already, so that people can get off-console or focus on more high-value tasks.
Engineering tools, operational logs, project management systems, team communications platforms, file backups, and data storage – these all need to be interoperable and accessible by the right people at the right time.
Epsilon3 brings such capabilities to the industry. The company has seen from experience that electronic procedures and MissionOps software can add enormous value to all levels of the space sector and are now bringing these solutions to the ecosystem around the world.
To find out more about Epsilon3, please view their supplier hub here on the satsearch platform.