WPA/WPA2 broadcast key rotation on a controller

This one is (IMHO) way beyond what you should need to remember for the exam, but I actually dreamed about it last night, so I though I should share the fun!
Special dedication to Kyle, he is the one who raised this issue and found the solution.... :-)
So here is the deal: when you use WPA or WPA2, your wireless client gets 2 keys: one unicast key, for its own traffic to and from the AP, and one broadcast key, which is a common key for all clients in the same cell. This broadcast key is used when the AP sends broadcast messages to all clients in the cell, so it's a shared key.
You may want to rotate those keys, change them every now and then so that a wannabee hacker would not have enough packets using a specific key to even dream of running an attack against your encryption scheme.
Changing the unicast key can be done by setting the EAP session, everytime the session is renewed, so is the unicast key... but the broadcast key is not, because it is shared. On the autonomous APs, you can rotate the broadcast keys from the Security > Encryption Manager page:

You can enable rotation at regular intervals. The gotcha here is that only 802.1X clients can join a network that has this feature enabled.
When using WPA, you can also enable key rotation "on membership termination" (every time a client leaves the cell, the key is rotated for the clients still in the cell), or on "membership capability change" (everytime a client using static WEP enters or leaves the cell, the key is rotated).

Ok, but what about the controllers? How to rotate the broadcast key on the WLC? There is no checkbox, and no CLI command for that... the key is rotated every hour... what? I want this feature for my clients security!!
Kyle found this nice info:

CSCsi27596—The controller lacks a supported way to configure the broadcast key rotation interval. Instead, it is hardcoded to a group key rotation interval of 3600 seconds (1 hour).

Workaround: On the console, configure the hidden command devshell dot1xUpdateBroadcastRekeyTimer(seconds). This command does not work in an SSH or Telnet session and does not survive a reboot.

Example:

(Cisco Controller) >devshell dot1xUpdateBroadcastRekeyTimer(86400)

value = 0 = 0x0

I dreamed of it because the example, instead of rotating the key every hour, seems to rotates it every... day (unless 8600 is the max value)! Wow, nice and a lot more secure!

I hope you don't need that for the lab, because this is pushing the system to its limit (and I hope you won't dream of it), but it is fun to know (or at least to know where to find it)...

:-)

WLC Layer 2 and Layer 3 Security Policies

In the Wireless Guest WLAN scenario, you need to create a WLAN on the foreign controller, and the same WLAN on the Anchor controller. You then add each controller to the other controller's mobility list (good practices dictate that controllers should be in DIFFERENT mobility groups. So they know each other, but they do not belong to the same mobility group). Last step, you define the Anchor as the Mobility Anchor for this WLAN on both the Foreign and the Anchor controller (don't forget to define the Anchor as the Mobility Anchor on the Anchor itself! This controller needs to know that it is one end of a EoIP tunnel).

That's the easy part, you can find tutorials on this config almost everywhere, for example at:
http://www.cisco.com/en/US/docs/wireless/controller/4.2/configuration/guide/c42mobil.html

Now, two questions:
1. Can you combine Layer 3 security (Web Authentication) with Layer 2 security?
2. What exactly needs to be the same and what can be different on the WLAN configuration on both controllers?

1. Yes! And you'd better do it if you want to secure the wireless part. With standard Layer 3 policy only, the only moment that is protected is when you access the Web authentication page and enter your credentials: at that time, you use HTTPS, so you are safe. As soon as authentication is complete, you use the pure Open WLAN, no encryption no encryption. Anybody sitting next to you can eavesdrop what you send and receive with no restriction... haven't you try already when waiting for your flight in an Airport lounge? Well, don't, it is completely in clear and fun to watch, but watching is completely forbidden in most countries...
So you can protect the traffic between the wireless client and the AP by adding to the Layer 3 policy a Layer 2 policy... but not in any conditions. For example, if you try to add WPA/802.1X and click Apply, you get this:

Well, we expected that! If you think about it, 802.1X is a user authentication that occurs at Layer 2... if this worked, you would have to start an 802.11 authentication/association process, get authenticated at Layer 2 with a strong mechanism (PEAP, FAST or even EAP-TLS!), then have to re-authenticate at Layer 3... not very handy and not very logical (why do you need this Layer 3 authentication if we already know who you are?)... and single sign on is out of reach for now...
So you can add a Layer 2 protection mechanism, but it has to be ENCRYPTION, not AUTHENTICATION. In other words, you can use WEP, WPA or WPA2, provided that you rely on PSK, not 802.1X. And then it works. This is how it looks:

Details of Layer 2 page:

And Layer 3 page:

Not very practical for public networks, as you would have to distribute the PSK to your Wireless LAN guest users, but fun as a configuration challenge.

2. Everything on the WLAN has to be strictly identical on the Foreign controller and on the Anchor controller, except:
- the Interface: you can send your users to a DMZ dynamic interface on the Anchor controller to the Management Interface on the Foreign controller (it is actually going to be the Management interface whatever your configured interface is on the Foreign controller, as Controller to controller communication always use the Management interface), so this can be different, and the Foreign controller does NOT need to (and should not) be configured with the DMZ dynamic interface you create on the Anchor.
- RADIUS server: in the WLAN Security > AAA Servers tab, you Anchor controller can define specific RADIUS server(s) to use, which your Foreign controller does not care about. Authentication is done on the Anchor, not on the Foreign, so you can call RADIUS servers on the Anchor and not on the Foreign, no problem. This can also be one difference.
- Profile Name: when you configure the WLAN, you setup a Profile Name and an SSID. The Profile name is a local identifier and not exchanged between controllers, so it can be different on the Foreign and the Anchor.
- Encryption key! Yep, that one is funny. One of my friends (thanks Amir!) discovered that 2 weeks ago. If you add Layer 2 policy with WEP, WPA or WPA2, you need to define a PSK... the PSK can be different on the Foreign and the Anchor! Now... which one will be used by the client? The one on the Foreign! Yes, this is the exception, everything comes from the Anchor controller, except this key. When you think about it, this makes logical sense. Your Layer 2 connection is to an access point connecting to the Foreign Controller, and you need to get to the Foreign controller before being sent through EoIP to the Anchor. So the Layer 2 protection is between you and the AP (connecting to the Foreign). It is therefore very logical to think that the key you need on the client is the key that the Foreign controller sends to the foreign AP!
Notice that you MUST define a key on the Anchor if you define a key on the Foreign: in the mobility exchange, they do tell each other that there is a Layer 2 key, and of which type (WEP, WPA or WPA2). It is just that they do not actually send that key, so it can be different on both controllers...
Wireless is so much fun!

Non-routed WLAN

You probably know the standard Guest WLAN scenario, with an anchor controller. You find it in most WLC configuration guides since code 3.2.

In this scenario, the wireless user is using a Web Auth WLAN and associates to an access point connected to the Foreign Controller. As you configure a Mobility Anchor on both the Foreign and the Anchor for the Web Auth WLAN, the wireless user is sent to the Anchor controller, gets its IP address from there, and starts its IP journey from the Anchor. In this classical scenario, the wireless user is limited to Internet access: the wireless laptop gets an IP address in the DMZ scope (10.10.10.0/24 in this example), from a DHCP somewhere in the DMZ. The Firewall is its gateway, and only allows traffic to the Internet, forbidding traffic back to the internal network.

Fine. But what if the network design is like this?

In this design, your firewall is integrated  into your edge router, and the "DMZ" just relies on subnets on the main switch.

In this configuration, your mission, if you accept it, is to design a non-routed WLAN... what is that? A config that will prevent the wireless user, sent to the Anchor controller, from getting to your corporate network... no ACLs allowed on the main switch...

This is how it works. You still create the same WLAN on the Foreign controller and on the DMZ controller, put them on the same mobility list (but in different mobility groups!), and set the Anchor as the Anchor controller.

What is different here is the IP assignment. Your must configure your main switch only with Layer 2 awareness of the DMZ VLAN. So if DMZ is network 10.10.10.0/24, VLAN 10, create VLAN 10 on the main switch, but not the Layer 3 SVI interface. As the main switch does not know subnet 10.10.10.0/24, it cannot route this subnet. The link from the main switch to the corporate network, configured as trunk, would allow all VLANs except vlan 10. For example:

conf t

vlan 10

interface g3/1

description --- to internal network

switchport trunk encapsulation dot1q

switchport mode trunk

switchport trunk allowed vlan except 10

Perfect. Now how does this wireless client get an IP address? The easiest way in this simple scenario (you saw that we even removed the DHCP server from the DMZ) is to provide the IP address from the DMZ controller. This implies 3 elements:

1. When you create the interface associated to the WLAN on the Anchor controller, the DHCP server IP address is the Anchor Controller Management IP address, like this:

The DMZ controller obviously has an IP address it this subnet. So, again, the DHCP server IP address is the controller Management Interface IP address (not the VLAN 10 controller dynamic interface IP address).

2. The gateway is the edge router, that is supposed to have an IP address in this DMZ subnet (10.10.10.254 in this example).

3. Create a DHCP scope on the Anchor controller for this 10.10.10.0/24 subnet.

This is it!

Written questions for the CCIE Wireless lab

"Effective January 4, 2010, the CCIE. Service Provider, Storage, and Wireless
Lab Exams will add a new type of question format in a section called Core
Knowledge. In this new section, candidates will be asked a series of four
open-ended questions which require a short written response be entered into
the computer--typically several words. The questions will be randomly drawn
from a pool of questions on topics eligible for testing. Candidates can
review the topics by visiting the CCIE track information on Cisco.com or
Cisco Learning Network. No new topics are being added as a result of this
change. Candidates will have up to 30 minutes to complete the Core Knowledge
section and may not return to it once they have moved on. A passing score on
the Core Knowledge section is required to achieve certification. Core
Knowledge questions were implemented on Routing and Switching labs in
February 2009, Security labs in June 2009, and Voice labs in July 2009, and
allow Cisco to maintain strong exam security and ensure only qualified
candidates are awarded CCIE certification. Candidates with exam dates
January 4, 2010 or later should expect to see the new question format on
their lab exam."

The aim of these 4 questions is to ensure that when you take the lab, you really know your stuff (Wireless in this case of course), and did more than just learn by heart series of click to answer a specific configuration question... so you will be asked 4 questions about wireless, and will need to answer in the form a a short paragraph or a few words, showing that you know what this is about. Of course, NDA prevents from giving real questions (and I have NO information anyway about these questions, that are kept very secretly as you can guess), but they could be questions like "In Wireless > 802.11a >RRM >DCA page, what does the "Avoid Foreign AP Interference" feature achieve?"... You are given a box where you can type your answer freely. It can be something as short and simple as "detect neighbor APs and try to avoid their channels when changing our APs channels", to something a lot longer and detailed, it is up to you. You just need to show the proctor that you know what this is about and give a simple (but still clear and precise) explanation, you do not need to write a 50 page white paper on the feature.
You have 30 minutes to answer all 4 questions, which is more than enough. You NEED to have most of them right to pass. Basically, you can have one wrong. More than one wrong, and you fail on your lab, even if your lab config is perfect...

Mobility group / list / domain RF Group

At the beginning of the world, wireless networks were small: one controller and a few access points. Then, the young Airespace company (some of you may even remember the even older name, Blackstorm Network) started having larger clients, needing several controllers. Some of them wanted to have controllers isolated from one another, some others wanted a "cluster logic" between controllers, to create a larger virtual controller.
So the Airespace team created the Mobility Group concept, also called Mobility Domain (these two terms mean exactly the same thing). If two controllers belong to the same mobility group, they exchange information about clients (namely, when a new client connects to a controller, this controller informs the other controllers. The result is that when the client roams to another controller, the new controller knows if it is a "new" client (no controller reported this client before), or a roaming client (one controller in the group/domain reported this client before). If it is a roaming client, you can even know which controller the client is coming from, which is very handy to transmit the client credentials from one controller to the other. You could have 12, then later on 24 controllers in the same mobility group/domain.
To make two controllers members of the same mobility group, you needed two steps:
1. Input the same string in the Controller > General > Mobility Domain Name field, so that both controllers have the same mobility group/domain value.
2. Inform each controller about the other, in Controller > Mobility Management > Mobility group. Each controller needs to know the other controller's Management IP address and built-in MAC address.
As a side effect of roaming, we also used to say that if you want roaming to occur smoothly, you'd better run the same code on both controllers (so that they speak the exact same language), and also configure the same virtual gateway IP address (this address is used as a "virtual address" to make the client think that it connects to one big virtual controller instead of several physical controllers).

At that time life was simple... :-) Then some clients also asked to segregate RRM... this was their scenario (kind of): I have 3 controllers, 2 in my office building and one in my warehouse. I want roaming between all of them (okay, so same mobility group/domain and controllers know each other) BUT I do not want cooperation for RRM between the office building and the warehouse. Why? Basically because the warehouse is a specific environment, most of the time isolated from the office and it has specific settings, so I do not want a sort of global master of the network that would not be able to distinguish the warehouse environment from the office environment.
Fine! We are going to create another group concept, the RF-Group (defined in Controller > General). You can put one string for the office building controllers, and another string for the warehouse controller. IF the strings are different, the controllers won't work together for RRM management...
Ok, this is how it works: when you add controllers to the local Mobility group/domain, those controllers send an introduction message to each other (hey, I am X, and oh BTW, my RF-Group is Y). All members of the mobility group also having a common RF-Group value elect an RF-Group leader. The RF-Group leader decides of an RF-Group hash (that represents this RF-Group name shared by those controllers) and sends it back to all members of the RF-Group.
Then, each AP sends, every 60 seconds, a RRM neighbor message from its radios, on all serviced channels. This message contains, among other things, the RF-Group hash. Neighboring APs hearing this message forward it to their respective controller. The controller looks carefully at the message and reads the RF-Group hash value. 2 possibilities:
1. The read RF-Group hash value is different from the RF-Group hash known by the local controller (so the other controller is part of another RF-Group, or the controller to which the AP is connected is unknown to the local controller RF-Group leader): the local controller despises and proudly drops the RRM neighbor message (this is from the RRM standpoint, your good controller may send alerts about rogues etc, but it does not see the neighbor as an RRM partner).
2. The read RF-Group hash value is the same as the RF-Group hash known by the local controller. In that case, the local controller thinks "hey, we are part of the same gang! We should work together on this RRM thing." The controller writes downs which of its access points hear which other controllers access points, and forwards this information to the RF-Group leader. Every 600 seconds by default, the RF-Group leader sends its instructions to the members of the RF-Group: you do this, you do that.

So you can clearly see here that the RF-Group is thought as a sub-group of the mobility group/domain. The other controller needs to be known to the local controller in order to be part of the same RF-Group and gets the same hash... and nothing in the RF-Group configuration allows you to tell the local controller about the other controllers, so it has to be done through the Mobility group/domain. So you configure Mobility group members, and among them some have the same RF-Group value and also share RRM information... limitation was 20 controllers part of the same RF-Group (so 24 controllers in the same mobility group, but yes, 20 members of the same RF-Group max).

Then some more complex scenarios (and more demanding clients?) appeared. They said: your stuff is cool, but does not work as it is, for 2 reasons:
- I want to roam across more than 24 controllers (I have a supabig network)
- I want APs in 2 mobility groups to exchange RRM information. For example, I have 2 floors, 1 controller per floor; people are not roaming from one floor to the other (coz they don't fly through ceilings), but the APs hear each other. So I don't want controllers to exchange information about clients (because it wastes bandwidth as these people will never roam from one controller to the other), but I do want them to exchange RRM information, so that they do not stupidly stay on the same channel.

Okay, it was time to extend the system... and the mobility group/domain concept became the mobility list... the idea is that your controller can know guys with the same Mobility group/domain value, and other guys with another Mobility group/domain value. Your list can contain up to 48 names in controller code 5.0, and 72 on 5.1 and later (so in the CCIE exam, we are still for now on code 4.2, with one mobility group of 24 members max). The key concept is that you do not need to have the same mobility group value to roam. As long as controllers know each other (they are on each other mobility list), they exchange information about connecting clients. The mobility list is what where we used to configure mobility group members before, defined in Controller > Mobility Management > Mobility groups (notice the plural form now).

So what is the difference? Do we still need to care about the Mobility group/domain value? Well, yes! There is a huge difference between mobility group (guys having the same mobility group/domain name as yours) and mobility list (other controllers you know, but that have a different mobility group/domain name from yours): CCKM (Cisco Centralized Key Management) and PKC (proactive Key Caching) do NOT work across mobility groups. This means that if you roam to a controller you know but that has another mobility group value, everything is fine is you use pre-shared key (or open or Web authentication). If you use 802.1X with CCKM or WPA2, your key will not get transmitted to the other controller... and the practical result is that you will have to re-authenticate to get a new key. You will keep your IP address though (as part of the roaming process), so the effect is that you will be briefly disconnected, but your IP session will not be broken. So this is fine if you are a data device. If you are a VoWLAN device, you do not want this disconnection, and you will make sure that controllers you roam to belong to the same mobility group.
So there is still roaming across mobility groups: when you get to new controller, it recognizes you as a valid client coming from another controller, and accepts you. It just fails getting your key if your use CCKM or PKC.

The RF-Group remains unchanged: all members of the same mobility list exchange introduction messages. Those that share the same RF-Group value elect a group leader and start working together on RRM... simple, isn't it?
:-)