Comparison of LLMs with networking tasks
Create a working, fully functional production grade Cisco IOS-XE 17 configuration of a router ISR8000 which can be copy pasted into a production device.
Properties
Add +1 point for:
Extra credit for identfying configuration issues, ie. IPv6 syslog server without any IPv6 routing, address, etc.
| Model | IP/VLAN | BGP/Traffic Eng. | OSPF | NAT | Shaping | Sec/AAA | Total |
|---|---|---|---|---|---|---|---|
| GPT-5.3-chat | 1 | 0.5 | 1 | 0 | 0 | 1 | 3.5 |
| GPT-5.5 | 1 | 1 | 1 | 1 | 1 | 1 | 6.0 |
| Opus 4.7 | 1 | 1 | 1 | 0 | 1 | 1 | 5.0 |
| Qwen 3.6-Plus | 1 | 1 | 0.5 | 0 | 0 | 1 | 3.5 |
| Gemini 3.1 Pro | 1 | 1 | 1 | 0 | 1 | 1 | 5.0 |
| Gemma 4 31B IT | 1 | 1 | 1 | 0 | 0 | 1 | 4.0 |
| Sonnet 4.6 | 1 | 1 | 1 | 1 | 1 | 1 | 6.0 |
These two models provided the most complete “copy-paste” production configurations.
192.168.0.0/28), NAT is required for internet access.local-preference for outbound steering and AS-Path Prepending for inbound steering, ensuring WAN1 is the Primary path.policy-map shaping to match the 50Mbps and 20Mbps upload constraints.crypto key generate is an EXEC command and cannot be part of the config-paste block, providing a separate instruction for it. GPT-5.5 correctly noted the IPv6 syslog reachability issue.passive but didn’t include a network statement or ip ospf area command that would actually allow a neighbor to form on that segment if it were changed to active.Most models correctly identified the IPv6 Syslog Paradox:
The prompt asks for a SYSLOG server at
2000:DEAD:BEEF::1, but the provided interface requirements are strictly IPv4. Without an IPv6 address on the WAN/LAN or a tunnel, the router cannot route the syslog packets.
Sonnet 4.6 and GPT-5.5 were the most vocal about this, with GPT-5.5 explicitly adding a “Note” that IPv6 reachability must be established for the syslog to function.
Common Miss: Only Sonnet 4.6 and GPT-5.5 addressed the Asymmetric WAN2 (100 Down / 20 Up). In Cisco IOS, the bandwidth command is mostly for metric calculation; to actually respect the ISP’s 20Mbps policing on the upload, a service-policy with a shape average 20000000 is mandatory to prevent packet drops.