Cisco: the four bullets, every question, every answer
Technical Intern, AI Infrastructure & Platform Systems, Jun to Sep 2026. This is the bullet an identity SWE will open first. Answers come from your two design docs. check marks a guess. Product names only in the room; no internal codenames.
The frame: say this before any bullet
What is Hyperfabric, in 30 seconds?
Answer
Cisco's cloud-managed data center fabric. You design a leaf-spine blueprint in a SaaS portal, the switches ship, get cabled, and zero-touch register with the cloud controller, which owns all the routing config so the operator never touches a switch. Monitoring is assertion-based: the controller continuously checks that reality matches the blueprint and latches warnings. The AI version adds GPU servers with ConnectX and BlueField NICs and monitors down to the NIC.What is the simulator and why does it exist?
Answer
A controller that provisions, upgrades and monitors thousands of customer devices needs a fake fleet to test against. The simulator runs each switch and server as a container, with the real production device agents (Go) running unmodified inside them, and a sidecar per server that mocks the BMC's Redfish API. Virtual cables are veth pairs moved into each container's network namespace, so link state, counters and neighbor discovery are real kernel behavior. The simulator itself is Python. I made the fake fleet more realistic (telemetry), more dynamic (live breakout), and closer to how customers actually install the agent (nested server mode).What was Go and what was Python?
Answer
Simulator, fixtures, templates and tests: Python. The device agents that collect data and talk to the controller: Go. I touched Go where a collector was brittle or had no test seam. The rule in the doc was "shim the device, change the collector only when it has to."Did you work on real hardware?
Answer
check Say it plainly either way. If simulator only: "No. My work was the simulator the controller and agents are tested against; the fixtures were validated against captures from real servers." Do not hedge.Line 1: temperature, power and health data for switches, servers and BMCs
Explain this bullet in 60 seconds.
Answer
A healthy simulated fabric still logged missing-data warnings from the agents and left the controller's view of fans, power supplies, temperature, firmware, optics and management interfaces empty. The agents read surfaces the simulator did not provide: Redis keys for sensors and transceivers on the switch, a firmware inventory file, platform commands (a show-platform command, an optics utility,nmcli for the management interface, mlxlink JSON and Mellanox ethtool -m for NIC optics), and Redfish payloads from the BMC. I treated each as a contract: what the agent reads, the parser shape it expects, the downstream type it fills. Then I added deterministic fixtures keyed by hardware model, plus command shims, so the real agents succeeded unchanged. Healthy baseline first, then fault scenarios.What is a BMC and why mock it separately?
Answer
A small controller on the server board with its own network path. It reports power state, fans, PSUs, temperature and firmware even when the host OS is down, over Redfish (HTTPS plus JSON) or the older IPMI. The collector must tell "host off, BMC reachable" apart from "both unreachable," so the simulator needs to produce both. That is why it is a sibling container reachable over the simulated network, not a file inside the server."Matching each device's data format to production collectors." What does that mean concretely?
Answer
I did not invent a wire format. The agent's parser is the contract. If a fixture has the wrong field name, unit, enum, or JSON shape, the parser drops it or misparses it and the test is worthless. So for each data family I read the parser, built a fixture that satisfies it for a healthy device, and wrote tests on both sides: parser tests on the fixture, and rendering tests that the right fixture is selected per hardware model and the shims land at the paths the agent calls.Why shim commands instead of changing the collector to tolerate missing data?
Answer
Making the collector tolerant would make the simulator pass while hiding what production needs. Simulating the command keeps the production code path identical, so a parser bug shows up in the simulator before it shows up on a customer's switch. Collectors were changed only where they were already brittle or had no test seam.How did you prove it worked?
Answer
A measurement harness before any fixture: aggregate warning and error tags from both agent logs, count the Redis keys for environmental and optics data, sample the downstream state, and produce a before-and-after report. Then layered tests: parser tests on fixtures; template rendering tests; a live smoke run (one switch, two server models, BMC sidecars, wait for agents to settle, assert zero missing-data tags); and scenario tests that degrade a component, verify the fault is reported through the normal path, restore, and verify recovery.What fault scenarios did you build?
Answer
Fan absent or failed, PSU absent or failed, temperature warning and critical, optics warning and alarm, management link down or no address, BMC sensor degraded. Each had degrade, verify, restore steps so the test could prove the alert fires and clears, driven from a small CLI.What was the hardest fixture?
Answer
check Candidates from your doc: the Redfish PSU sensor names on the UCS C225 (every BMC vendor names things differently), thenmcli multiline terse format, or mlxlink's JSON for per-lane optics. Pick the real one and say why it was hard and how you found the truth (a capture from a real server).How would you detect missing or garbage telemetry in production?
Answer
Treat absence as a signal: track last-seen per device and alert on staleness. Validate ranges (a 300 C inlet temperature is a bug, not a fire). Counters must be monotonic; a decrease means a reboot or a wrap, so compute rates on the collector with wrap and reset detection. Compare with peers: one switch 30 C hotter than its neighbors is a story. Use an explicit "unknown" state instead of a silent green; Hyperfabric's gray assertion state is exactly that.Security angle: a device sends data that looks valid but is malicious or spoofed. What then?
Answer
Two layers. Trust the channel, not the payload: each device authenticates to the controller with a device identity (the TPM-backed certificate path on servers, the claim-and-bind model on switches), so a payload is tied to a known device. Then validate the payload anyway: schema, ranges, monotonic counters, and cross-checks against the blueprint (a device reporting ports it does not have is flagged). The simulator fixtures are tagged per model and the launcher refuses unsupported combinations before anything starts, which is the same instinct: fail closed on unknown shapes.What would you alert on first?
Answer
In an AI fabric: link flaps, CRC errors, discards, optics receive power below threshold, temperature. RDMA traffic is supposed to be lossless, so any drop stalls GPU jobs. For a general service the equivalent list is error rate, staleness, and anything that changed when nothing should have.Line 2: live port breakout, 0% dropped traffic, no restarts
Explain this bullet in 60 seconds.
Answer
Breakout splits one physical port into several logical ones, for example one 400G port into four 100G ports. On real hardware that is disruptive to the port. In the simulator the equivalent cost was worse: child interfaces were created on demand, which restarted the simulated device container and dropped traffic on everything attached to it. I pre-created every possible child interface in the down state at boot and wrote a watcher that brings the right ones up when the controller requests a breakout. Nothing restarts, and the rest of the fabric never notices.How did you make it live?
Answer
check Every port that can be broken out gets its children created at boot, admin-down, in the device's network namespace. A watcher watches for the breakout request reaching the simulated device and, in one ordered step, marks the parent down, brings the children up, and starts neighbor discovery on them. The operation is idempotent: a retry finds the children already up and does nothing. Confirm the watcher mechanism (netlink events, polling, or a controller message) and interface types (veth or dummy).How did you measure 0% drops? 0% of what?
Answer
check An end-to-end test ran traffic (iperf3 and continuous ping) across the device's other links during the breakout and compared drop and error counters fromip -s link before and after; the delta was zero and the probes saw no loss. Say the tool and the duration you actually used. The claim is about the rest of the fabric and the device not restarting; the parent port itself stops existing as a single port, so say that before they catch it.Race: a breakout request arrives while the parent carries traffic, or two requests arrive at once?
Answer
check The watcher serializes operations per port and applies each as one transaction (parent down, children up), so there is no window with both active and no way to create duplicate children. A retry after a partial failure converges to the same state. In the simulator the cost of getting this wrong is a failed test; on a real switch it is an outage, which is why the real ASIC reinitializes the whole port group.Why is pre-creating interfaces a good design and not a hack?
Answer
It trades a little memory at boot for zero disruption at runtime, and it keeps the thing that holds state (the device container, its agents, its sessions to the controller) untouched. The general pattern is pre-provision, enable on demand, never restart the component that holds the connections. The same idea shows up in connection pools, warm standbys, and feature flags.What does the controller see after a breakout?
Answer
The parent disappears as one port and the children appear with their own names, link state, counters and neighbors. Assertions check the result matches the blueprint and the optics on the other end.Line 3: planned timeline cut 60% with AI coding agents, gated by tests and review
60% of what?
Answer
check The environmental-data proposal had a ten-week plan, one data family per week (switch firmware, switch optics and counters, server management state, server optics, BMC, fault scenarios). The work landed in about four weeks. Confirm the exact weeks and that the baseline was the written plan, not a guess after the fact.How did you split the work so agents could run in parallel?
Answer
Once each collector contract was written down, the data families were independent: separate fixtures, separate shims, separate tests. One agent per family, each with its own test target, so they could not step on each other. The shared pieces (fixture selection by model, the measurement harness) I wrote myself first.How did you keep agent-written code from being wrong?
Answer
Three gates, none optional. Unit tests on parsers and fixtures, written as failing tests before accepting a change. End-to-end tests against the real device agents, because a fixture that passes its own test but makes the real parser unhappy is worthless. Human review of every diff, which was the real gate. The agents proposed; tests and I decided.Give me one thing an agent got wrong that a test caught.
Answer
check Must be real. Your notes say the most likely shape is a shim that produced plausible output with fields in the wrong order or the wrong unit, caught by the parser test. Have the specific one ready, with what the test asserted. Do not use the 32-bit counter story unless it actually happened.What would you not let an agent touch?
Answer
Anything that changes a contract or a security boundary: the collector's parser, the rule that the simulated server never gets the outer Docker socket, the TPM state handling. Those were human decisions with a written review. Also anything I could not test automatically, because then review is the only gate and review alone is not enough. Say this maps directly to Agent ID: scope what each agent can touch, log what it did, verify before trusting.Isn't "60% faster with AI" just a buzzword line?
Answer
Agree that the number is only meaningful with a baseline and gates, then give both: the written ten-week plan, the four weeks, the three gates. Then the honest cost: review load went up, and you would not run this on work you cannot test. Skeptics respect the cost being named.Line 4: server support designed with 7 engineers (Linux, containerd, Podman, K3s, TPM, ConnectX/BlueField)
Explain this bullet in 60 seconds.
Answer
The simulator modeled a customer server as one container running the agents directly. A real customer installs the agent from an RPM or DEB on Ubuntu, Red Hat or Debian; the installer detects the container runtime (Docker, Podman, containerd, CRI), a systemd service verifies and imports a signed image and launches the agent container with host networking, or Kubernetes runs it as a DaemonSet. None of that was exercised. I proposed and led "nested server mode": an outer container that behaves like the host OS, with systemd as PID 1, running its own private container runtime that launches the real agent image through the real installer. containerd first, then Podman on a Red Hat base, then K3s for the Kubernetes path, Docker last. The simulated server never gets the outer Docker socket. Hardware identity (server model, NICs, TPM, BMC) is a separate profile from the execution environment (OS, runtime, deployment). Seven reviewers across software, networking and hardware.Why not mount the host Docker socket? It would have been much simpler.
Answer
Because a simulated customer server would then control every other container on the host: stop a switch, read another server's TPM state, start anything. We rejected it outright, ran a private runtime inside the server container, and wrote a negative test proving the outer daemon and sibling containers are unreachable from inside. Say "blast radius." It is the same instinct as least privilege: give the component only what its job needs.Why not a VM per server?
Answer
Best kernel fidelity but far heavier, and hard to wire into the laptop-based simulator and its virtual network. We wrote it down as the fallback for when a kernel-specific need appears, instead of paying for it up front.Why K3s and not kind or k3d?
Answer
kind and k3d add a node-container layer that hides the host OS behavior we wanted to test. K3s with its bundled containerd runs the agent as a DaemonSet with host networking on the simulated host itself, which is how a customer would run it.Why separate the hardware profile from the execution profile?
Answer
Hardware identity (serial format, BMC model, NIC adapters with PCI addresses, vendor and device IDs, driver, firmware, interface names, optics contract, TPM presence) changes independently from how the agent runs (OS and pinned image digest, package format, runtime and version, standalone vs Kubernetes, resource limits). Named, validated combinations fail fast before any container starts. A free-form "number of NICs" knob was rejected because it cannot keep PCI functions, Linux names, optics fixtures and BMC inventory consistent with each other.What is the TPM for, and what did you actually build?
Answer
Device identity. The agent uses the TPM to generate the key and certificate signing request for enrolling the device with the controller, so possession of the device is proven before configuration is pushed. In the simulator it is a software TPM with one persistent state store per simulated server, reached over a local socket through the TPM resource manager. State survives agent replacement and restart, is removed on clean, and is never shared between servers. It is a working TPM for identity flows, not hardware attestation. Say that boundary out loud.A gotcha you hit with the TPM?
Answer
The installer checks for a TPM device under sysfs, but a socket-based software TPM creates no sysfs device. Faking an empty device entry would make the device-transport code try to open a nonexistent/dev/tpm. So the launcher takes an explicit simulator setting for TPM presence and transport instead of pretending. Lesson: do not fake a signal you cannot back up all the way down.Two DMI views: what was that about?
Answer
The host-side installer reads the machine identity from the real sysfs DMI path before the agent starts, while the agents read a generated tree. The same product identity has to appear in both places or the installer and the agent disagree about what machine they are on. Identity consistency across two readers of "who am I" is a small version of the problem this team works on.ConnectX vs BlueField, and why does an identity SWE care?
Answer
ConnectX is a NIC. BlueField is a DPU: the same NIC plus Arm cores, its own OS and its own BMC, so it can own the host's network stack and show up to the fabric as its own managed endpoint with its own identity. Whether "BlueField support" meant the host-visible NIC view or an independently managed DPU was an explicit open question in the review. BlueField-4 was deferred because it pairs with a different NIC generation and there was no sample. No public emulator exists for these NICs, so driver, firmware and RDMA work stays on physical hardware.What is the fidelity limit of the nested design?
Answer
The OS profiles share the outer kernel. They validate packaging, systemd, runtime and installer behavior, not distro kernels, modules or drivers. Also no PCI devices or NVIDIA drivers are emulated; NIC identity is projected through generated sysfs and DMI trees plus command shims. Saying the limit out loud is what made reviewers trust the rest.How did you get seven engineers to agree?
Answer
Wrote the proposal so each reviewer could read only their part: executive summary, a decisions-at-a-glance table, a reading guide by focus area (architecture, runtime and packaging, hardware fidelity, Kubernetes, security, delivery), non-goals, and fourteen numbered open questions grouped by theme. Made the ask small: approve the architecture and a feasibility gate (boot systemd with writable cgroups, start containerd without the outer socket, import, start, stop and remove a pinned image, persist across a restart, expose the virtual interfaces, clean up), not the full hardware matrix. Then four rollout phases. Recorded every decision in the doc.Which question was contested, and how was it resolved?
Answer
check Candidates from your notes: rawctr as a supported interface, a UBI image versus full RHEL for Podman, K3s in the first milestone, or host-visible BlueField-3 versus an independent DPU. Pick the real one, give both sides fairly, say what settled it (a prototype, a customer requirement, a cost), and the decision. If it went against you, say so.How does networking cross the nested boundary?
Answer
The simulator's virtual cables are placed in the outer server-host namespace, and the agent container runs with host networking (hostNetwork in the Kubernetes mode), so it sees the same kernel interfaces. One risk recorded: runtimes create their own interfaces (docker0, a CNI bridge, a flannel VXLAN interface under K3s), which the collectors must not mistake for fabric ports, so the design has an unmanaged-interface policy and filtering.Identity and security angles they may take on any bullet
Where is identity in this project?
Answer
Three places. Devices prove who they are before the controller trusts them: the claim-and-bind model for switches and the TPM-backed enrollment for servers, which I simulated with a software TPM per server. Telemetry is trusted because the channel is authenticated, then validated anyway. And the no-Docker-socket rule is least privilege for a component: the simulated server gets only what its job needs.How would you threat-model the simulator?
Answer
STRIDE quickly. Spoofing: a simulated device presenting another's identity, prevented by per-server TPM state that is never shared. Tampering: fixtures altered to hide a fault, caught by deterministic fixtures under version control and scenario tests. Information disclosure: one server reading another's state, prevented by no shared socket and separate namespaces. Elevation: the Docker socket case. Denial of service: a runaway nested runtime, bounded by cgroup limits in the execution profile. Repudiation: the review doc and test reports record what changed and why.What secrets existed and how were they handled?
Answer
check TPM state stores per simulated server (never shared, removed on clean). Signed agent images verified by the installer. Any controller credentials the agents used: confirm whether these were per-device certificates issued at enrollment (the expected answer) and that nothing was hard-coded in fixtures or templates.Connect this to Entra in one sentence.
Answer
"A fabric where every device must prove its identity before it gets config, and every agent must be scoped to what it may touch, is the same shape as workload identity and Agent ID; I have built the toy version and want to learn the real one."Extra stories (drafts to confirm)
A simulated device that was "unreachable."
Answer
check The controller showed a server missing. Layer by layer: the container was up; the virtual interface was in the namespace but admin-down because the startup script ran before the virtual network attached it; brought it up, neighbor discovery appeared, the controller's assertion went green. Lesson: "missing" in a controller is usually a layer below the one you first suspect.Empty counters that looked like a dead port.
Answer
check The counters collector warned because the counter maps existed but were empty. The real question was "what does healthy look like": non-zero traffic and zero errors. I populated minimal realistic counters, kept errors at zero, and made the fault scenario set errors. Zero traffic is a story, zero errors is normal.The management interface shim.
Answer
check The server-side collector shelled out tonmcli, which the server containers did not have. I wrote a model-aware shim that returns the management interface in the multiline terse format the parser expects, generated from the configured management port. Good example of "the parser is the contract."What would you do differently?
Answer
Write the collector contracts down before touching fixtures (learned after the first data family), and build the measurement harness in week one, not week three, because "fewer warnings" is only a result if you can count them.What did you learn about working at a big company?
Answer
A written proposal with explicit open questions gets decisions faster than meetings. Reviewers trust you more when you state the limits of your design yourself. And production code paths are sacred: fake the environment, never loosen the thing you are testing.Glossary, one breath each
BMC out-of-band server controller · Redfish / IPMI its HTTPS/JSON API and the older binary one · DMI the firmware's product identity table, read from sysfs · LLDP link-layer neighbor discovery, how the controller learns cabling · breakout split one physical port into N logical ports · DOM optics diagnostics (power, temperature, voltage per lane) · Redis state DB where the simulated switch keeps its platform state · netns / veth Linux network isolation and the virtual cable between namespaces · systemd as PID 1 / cgroup v2 what a container needs to run its own services and child containers · containerd / Podman / CRI container runtimes and the interface Kubernetes uses to drive one · K3s / DaemonSet / hostNetwork small Kubernetes, one pod per node, pod shares the host's interfaces · swtpm / TCTI software TPM and the transport an application uses to reach it · CSR / enrollment the device proves its key and gets a certificate · claim and bind prove you own a device, then map it to a slot in the blueprint · assertion a continuously evaluated statement about expected state that latches on failure · DPU NIC plus Arm cores plus its own BMC · ZTP boot, find the controller, pull config.