Dave Borncamp

Background

16+ years building distributed systems, ground systems, and integrating Flight Software while leading engineering teams in defense, space, intelligence, and academic domains.

Education

• 2011 - BA Astronomy @ CU Boulder
• 2013 - AS Computer Programming and Geographic Information Systems @ FRCC
• 2016 - MS Computer Science @ Towson University
• 2021 - Graduate Certificate in Space Resources @ Colorado School of Mines
• 2022 - MS Space Resources @ Colorado School of Mines

Work

• 2010 - 2013: Southwest Research Institute

  • Research Assistant

• 2013 - 2017: Space Telescope Science Institute

  • Sr. Research & Instrument Analyst

• 2017 - 2021: Decipher Technology Studios

  • Solutions Team Lead

• 2021 - 2024: BAE Systems (Ball Aerospace)

  • Chief Software Architect

• 2024 - present: SciTec, Inc

  • Principal Engineering Manager

Technical Challenge

Controlling Message Release from a Kubernetes Cluster

• Context
• Approach
• Outcome

Context

• MDPAP is an operational missile detection, tracking, and warning system that is currently in use by the US Space Force.
• In an operational MDPAP system, data can be played back or simulated to test the system or review a previous event to compare it to what is currently happening.
• We needed to ensure that messages that were only in an `operations` state and were marked correctly would be released from the cluster to end users throughout the world.

Takeaway - 3 conditions needed to be met for a message release: correct instance state, simulation tag, and playback tag.

The Challenge

• There are MANY messages flying around the system and we have very tight latency requirements.
• If a bad message were to leave a system, there are many downstream consequences.
• The legacy system used adaptors that unpacked every message in every cluster to match the release requirements.

Solution Space - Kubernetes Network Policies

• By default pods are not isolated: all traffic in and out is allowed but a NetworkPolicy selects pods by label and lists the ingress or egress they are allowed
• Once selected, a pod only gets the traffic a policy allows in that direction.
• The CNI plugin enforces policies, so they do nothing without one that supports them.

• By implementing Network Policies we could check initial conditions for state without unpacking the messages which saved a lot of processing time and latency in the system.

Approach

• We implemented message gateways that are basically proxies which have external connections.
• The gateways would check the data packets for simulation and playback data on arrival.
• A Kubernetes operator and CRD allowed us to control this all based on state of the cluster

Start From Default Deny

• Network policies are fairly easy to implement, but default denying everything cuts off all communication to the pod

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: communications
spec:
  podSelector: {annotations...}
  policyTypes:
    - Egress

Allow the Release Path

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-message-release
spec:
  podSelector:
    matchLabels:
      app: message-release
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            ranges
      ports:
        - protocol: TCP
          port: port

Once the conditions are met, the CRD will allow the pod to communicate via the policy.

Outcome

• This allowed us to reclaim ~10% of CPU power in the system to be used elsewhere.
• Ensured that we met our release requirements at a cluster level, even if we had code that tried to release messages, it was not allowed.
• Communication Pods had to be restarted to ensure that long lasting connections like sockets are terminated.

Problem Solving and Skills

• The problem was easier solved by breaking out the state into the cluster level and the data tagging at the message level.
• By removing one of the problems and solving for that condition first, efficiencies can be gained.
• Future work:
• Consider breaking out metadata from the message to make it quicker to unmarshal.
• Hot load the pods so that they do not need to be restarted.

Sierra Space Values

We Are One Team

Describe a time when you contributed to fostering a strong sense of teamwork and collaboration within your team.

• Whiteboarding sessions
• The "thank you" totem
• Team shoutouts

The Team

• The team is 11 engineers spread across multiple timezones and 3 companies.
• Each area and company had their own culture and way of working.
• I needed the team to be cohesive, with active participation from everyone to get the best product.

Whiteboarding Sessions

• Ownership is key to long term success.
• Allowing the engineers to come up with key pieces of the design gives them that agency that is needed for real ownership.
• Whiteboarding sessions allow the engineers to come together and discuss possibilities and dig into the design, rather than be told to implement something
• It also allows for communication that is technical which can help form the basis of a real bond between people.

The "Thank You" or "Great Work" Totem

• Recognition is a key piece to motivating people, but it doesn't have to be complicated or expensive.
• Expressing gratitude to others on the team helps form a better bond and builds good habits and bonds.
• A totem (in this case, a simple clay wizard) is passed to another person during a scrum at least once a sprint with the giver saying why the next person deserved it.
• These could be small things, or major things. The idea is to just get people to interact more in a positive way.

TODO: Photo of the totem

Team Shoutouts

• A simple kind word can motivate people to move mountains, and positive reinforcement is often lacking from the workforce.
• An email or acknowledgement during a meeting can mean the world to some people.

What Changed

• Engineers from multiple companies can be frequently seen talking and being friends.
• Collaboration across timezones is much higher because people feel safe in reaching out and know the other people on their team as people.