poweringdiscovery.automate-the.net

Powering Discovery

An expedition log from the trail: the network refresh, the remote rollouts and the segmentation program that got me here, and why that route leads to NERSC's Network Engineer, Platform, Automation & HPC/AI role.

Claudia de Luna · Scroll to climbPress → to climb

Trailhead

Nobody starts at a high camp

The story of the last few years is a network segmentation program. But segmentation could only begin once there was a modern network and OT/IoT monitoring in place, and a lot of climbing went into reaching that start line.

YOUR GUIDEA snow leopard who knows this terrain, walking the route with you and pointing out what mattered at each camp.
THE OWL · OBSERVABILITYKeeping watch over every camp, built on SuzieQ. Look for the owl's eye at every stop.

Stage 01 · Base Camp

How we got to the starting line

01.A

Data center and network modernization

Data center and network refresh: assessments, BoMs, inventory and zero-touch provisioning, with quality assured without freezing changes.

01.B

OT/IoT security platform, mid-Covid

No travel. About 200 cybersecurity platform appliances installed by on-site resources, guided by a one-page installation document.

01.C

Firewall appliances

Once the client saw we could deploy hardware without traveling, they had us install their firewalls too.

Stage 02 · Camp I

Ready for segmentation… or were we?

This is the heart of the program. Before a single zone could exist, it had to answer four questions. Open each one to see how we did it.

02.A

The previous solution

What do we do with the previous segmentation solution?

02.B

Which devices, where

What devices are at each location, and which ones need to go into a security zone?

02.C

Zone design

How many zones, how big, and what policy for each? 1,107 zones across 185 locations.

02.D

Provisioning across silos

How do we provision security zone VLANs across a highly siloed, fractured IT organization?

Stage 03 · Camp II

Handing the mountain back to the site

How does a site operate in a newly, highly segmented environment? The program left each site with a slide deck and URLs. After handoff we started to see drift: devices moving back out of their zones, and new devices that belonged in a zone never put there. So we recommended three next steps. Only one was adopted, and only in a very stunted way.

03.A

Continual auditing

Keep checking which devices are in which zone, so drift is caught instead of discovered.

Recommended · not adopted

03.B

User self service

Let the people closest to the devices see and act on zone membership themselves.

Adopted, in a very stunted way

03.C

Provisioning as a service

Turn zone provisioning into a repeatable service, not a cross-silo negotiation every time.

Recommended · not adopted

The fixed rope · Observability with SuzieQ

The owl sees the whole network

Quiet, watchful, always looking. Observability was critical at every camp, and it ran under the whole climb like a fixed rope.

More on the owl: No Rest for SuzieQ and My Network is Talking Back Thanks to SuzieQ MCP.

  1. During refresh

    Pre and post validation: proof that everything came back.

  2. During segmentation

    Getting devices onto the network, finding where they were, and seeing whether they moved into or out of a zone.

  3. During provisioning

    QA on every zone that was provisioned.

  4. After handoff

    The foundation of user self service.

Kit check

What was in the backpack

Lessons learned · A preview of my AutoCon6 talk

If I climbed it again

Coming to AutoCon6: My network automation do-over using the NAF Framework. These are the lessons from this trail that shape it.

LESSON 1

Decide the old route's fate up front

WHAT HAPPENED  The client kept its VRF-based solution, so the site had two segmentation solutions to support and the IT teams had two to operate.

DO-OVER  Make retiring, or integrating, the existing solution part of the program's scope from day one.

LESSON 2

Capture policy intent before the firewall does

WHAT HAPPENED  Zones started with "allow any/any on the intranet" and logging on; the real policy came from mining firewall logs. Far more manual than it had to be.

DO-OVER  Declare each zone's intended policy as data, and use automated log analysis to confirm it, not to discover it.

LESSON 3

Get the inventory out of Excel

WHAT HAPPENED  Three good sources (observability, Cisco ISE, OT monitoring) ended up in a spreadsheet, and the community detection algorithms we planned never ran.

DO-OVER  Collect into a source of truth that tools, and algorithms, can actually use.

LESSON 4

Automate the checks, not just the push

WHAT HAPPENED  Config push wound up not being used much. Ticket creation, payloads, rollback configs and QA carried the program.

DO-OVER  Start with validation and QA, then automate the change itself.

LESSON 5

A handoff is not a slide deck

WHAT HAPPENED  Each site got a slide deck and URLs. Of three recommendations, only self service was adopted, and only in a very stunted way.

DO-OVER  Design the operating model, and adoption, into the program, with the people who will run it.

LESSON 6

Drift is guaranteed

WHAT HAPPENED  Devices moved back out of their zones, and new devices never got zoned.

DO-OVER  Compare observed state to intent continuously, from the first zone onward.

The do-over, block by block

Rebuilding the program on the NAF Framework

The Network Automation Forum's framework gives the do-over its structure: the same program, one building block at a time.

Intent

ON THIS TRAIL  An Excel data set, tickets, firewall logs

DO-OVER  Zones, membership and policy as structured, version-controlled data

Collector

ON THIS TRAIL  Three sources merged by hand

DO-OVER  Automated collection from observability, Cisco ISE and OT monitoring

Observability

ON THIS TRAIL  QA and drift checks

DO-OVER  Continuous comparison of observed state to intent

Orchestration

ON THIS TRAIL  Ticket creation and payloads

DO-OVER  Provisioning as a service, end to end

Executor

ON THIS TRAIL  Config push and rollback configs

DO-OVER  Changes generated from intent, with rollback built in

Presentation

ON THIS TRAIL  A slide deck and URLs

DO-OVER  Real self service for the people at the site

Camp III · Intent

Intent is the map

Every problem on this trail was a gap between what someone intended and what the network actually did. The focus should be on intent, captured so that it is programmatically consumable.

Intent

What should be true: zones, membership, policy.

Automation

Make it true: tickets, payloads, change, rollback.

Network

What is actually true, today.

Observability

Check it: does the network match the intent?

The guide holds the map; the owl watches the terrain. The do-over keeps them talking, and the loop never stops. Diagrams become a view of intent, not the source of it. (I never want to use Visio again.)

Where intent lived on this trail

From spreadsheets and tickets to data

ZONE MEMBERSHIPAn Excel data set
PROGRAMMATICALLY CONSUMABLEStructured data in a source of truth
ZONE REQUESTSTickets carrying config payloads
PROGRAMMATICALLY CONSUMABLERequests validated against intent before they become config
ZONE POLICYMined from firewall logs after "allow any/any"
PROGRAMMATICALLY CONSUMABLEDeclared per zone, then confirmed by the logs
QA"Were the requested zones configured on exactly the switches requested?"
PROGRAMMATICALLY CONSUMABLEThe same question, asked continuously
HANDOFFA slide deck and URLs
PROGRAMMATICALLY CONSUMABLEThe intent itself, which the site's tools can check against
ALREADY ON THE WAY · GRATUITOUS-ARP.NETMy Network is Talking Back Thanks to SuzieQ MCPAn AI assistant checks intent in Infrahub, verifies live state with SuzieQ, and pushes configuration with Netmiko, under supervision. ALREADY ON THE WAY · GRATUITOUS-ARP.NETNo Rest for SuzieQDial-a-Vlan: a small self-service app that checks state through the SuzieQ REST API before Netmiko changes a port.

Base camp at NERSC

Where I'd start at NERSC

Ideas, not assumptions: first I'd learn how NERSC runs today. But this trail suggests where to look.

1

Observability before change

Baseline the network before Data Center modernization and NERSC-10 change it. Pre and post validation as standard practice.

2

Intent as data

Capture the border, backbone and edge design as programmatically consumable intent, and generate the documentation and diagrams from it.

3

Automated QA on every change

The zone QA questions, applied to the 800G/400G backbone and the 1 Tb/s border: did the change do exactly what was intended?

4

Proactive, not "self-healing"

Catch failures that could impact science workflows before 10,000 users do, and automate the remediation that is safe to automate, as I did for DHCP scope exhaustion.

5

Built with the people who run it

Operations and the Security Group in from day one, so the result is adopted, not just delivered.

Summit · NERSC at Berkeley Lab

Where the trail meets the role

Network Engineer, Platform, Automation & HPC/AI (107394). Each camp on this route, and some from earlier climbs, maps to something the role asks for.

ON THE TRAILNetwork refresh and new Cisco ACI data centers: assessments, BoMs, zero-touch provisioning, and QA without freezing changesAlso on the route: Terraform for Cisco ACI.
IN THE ROLEContribute to, and lead, Data Center modernization for the Smart Facility initiative.
ON THE TRAILObservability at every camp, from pre/post validation to the foundation of self serviceAlso on the route: SuzieQ, including its MCP server. A device data set built from three sources (observability, Cisco ISE, OT passive monitoring), with community detection algorithms planned to help group the OT devices; in the end, the data set itself did the job.
IN THE ROLEImplement, then architect, automation and observability; build AI/ML-driven observability and predictive analytics.
ON THE TRAILPre/post validation, zone auditing and provisioning QA, automatedAlso on the route: JPL's rule: do something more than twice and you automate it. Python, Ansible, Nornir and Netmiko; some Terraform and eAPI.
IN THE ROLEAutomate the network's daily operational, deployment and monitoring tasks; design and maintain automation frameworks and infrastructure as code.
ON THE TRAILCatching drift: devices moving out of zones, new devices never put in oneAlso on the route: Automated remediation of DHCP scope exhaustion.
IN THE ROLEGet ahead of failures that could impact services, with automated remediation.
ON THE TRAILSecurity zone design and per-zone policy, worked side by side with securityAlso on the route: At JPL, a firewall policy tracking system built with the security team. Thorough failover testing before every delivery, client permitting.
IN THE ROLEDesign and implement network security architectures, including segmentation and access control, with the Security Group; redundancy and failover.
EARLIER ON THE ROUTEAt JPL, managed the edge and peerings with Caltech, NASA and Los Nettos/CalRENAlso on the route: Performance work at Barclays, and across a whole career.
IN THE ROLEMonitor and optimize performance: latency, throughput, fault tolerance; BGP and OSPF routing.
ON THE TRAILProvisioning across a highly siloed IT organization, and hardware rollouts with no travel during CovidAlso on the route: Both sides of the table, enterprise and consultant. Advisory Board of the Network Automation Forum and a contributor to the NAF Framework.
IN THE ROLEBuild working relationships across disciplines; collaborate with technical peers, vendors and customers; resolve complex issues in creative ways.
ON THE TRAILStanding up the segmentation program: what to do, who to bring in, which tooling and automation stackAlso on the route: Owned scope and budget since before project management was its own discipline.
IN THE ROLEEstablish technical direction and methods for new or ambiguous assignments; lead technical project scope, schedule and budget.
ON THE TRAILAssessments, BoMs and a one-page installation guide that on-site staff could followAlso on the route: Intent over Visio: documentation captured as intent that programs can consume.
IN THE ROLECreate and maintain network documentation, including physical and logical topology.

Next climb

Routes I haven't climbed yet

The posting welcomes non-traditional paths, so here is the honest map. Understanding how things function is the beginning of troubleshooting, and it is how I'll learn this terrain.

HPC fabrics

RDMA, InfiniBand and RoCE are new ground for me, and the growth area I'm most keen on.

Fabric management

NVIDIA UFM and Dell SFM are new; my Arista CloudVision experience is end-user level.

Protocols and cloud

No L3VPN yet, and my VXLAN and IS-IS so far is through Cisco ACI. MACsec is new. Cloud is AWS with Terraform, GCP in the lab, but no hybrid Direct Connect or ExpressRoute project. Python over Go, by choice.

Operations

An escalation contact in almost every role I've had. To be clear: I don't come from operations, and it's been a long time since I was in an on-call rotation.

Questions?

Your guide and the owl will take them from here.
THIS SITE
poweringdiscovery.automate-the.net LINKEDIN
linkedin.com/in/claudiadeluna BLOG
gratuitous-arp.net AUTOCON6
My network automation do-over using the NAF Framework