The following test procedures are to evaluate the Meraki Wireless cloud WLAN Controller (MCC) product. The primary goal of this evaluation is to determine if the Meraki system has the functionality required to operate in a Customer office environment. All testing will be performed in a lab that will simulate a Customer Product Development office environment. It should be noted that this testing was not an in depth evaluation and stress test of each feature. The goal is to determine if the product functions as advertised and whether it would easily fit into Customer's network architecture. 

The scope is to test the functionality and operation of the Meraki MCC. The goal of this testing is to verify that certain functionality/features called out in the Meraki Cloud Controller Product Manual (MCC-PM) operate as described.  The testing done was functional testing, not interoperability/integration testing, not in depth protocol analysis of each function or stress testing of those functions.

Test Objectives

 Meraki Basic Operation 

•   Create a network: PASS
•  Create SSID's: PASS
•  Statically IP address an AP: PASS
•  Obtain an IP address through DHCP: PASS
•  Client IP addressing through NAT mode (clients get IP's from MCC): PASS
•  Client IP addressing in Meraki bridge mode (enterprise users): PASS

· AP Configuration changes made on the MCC; 

• o How long does it take for the changes to be applied: PASS
• o Does the AP go offline: PASS
• o Does it affect attached clients: PASS

· MCC interface responsiveness 

• Verify Monitoring features
• All Network Overview page: PASS
• Maps Page: PASS
• AP Page: PASS
• Clients Page: PASS
• Traffic Analysis Page: PASS
• Client Details Page: PASS
• Event Log Page: PASS
• Rogue AP Page: PASS
• Summary Report Page: PASS
• Live Updates: PASS
• Search Tool: PASS
• Email Alerts: PASS
• Export XML Data: PASS
• Logins Page: PASS
• Account Activity Page: PASS

· Verify AP mapping capabilities: PASS

· Verify/Clarify Meraki optimization algorhythms/routines: PASS

· Verify Meraki network operation when MCC is down or unreachable: FAIL

· Verify Meraki mesh networking capabilities: PASS

· Over the Air Upgrades N/A

· Verify VPN's into a Customer environment work properly: FAIL

· Test Dot1q trunking capability

•  Per SSID VLAN Tagging: PASS
•  Per User/group VLAN Tagging (dynamic VLAN allocation): N/A
•  Verify management traffic: PASS

· User Access Control

•  MAC Whitelist: PASS
•  MAC Blacklist: FAIL
•  Bandwidth Shaping: FAIL
•  Firewall Rules for RF Users: N/A
•  LAN Isolation: PASS
•  Custom Firewall Rules: N/A

· Identity Policy Manager: PASS

· Traffic Shaper: FAIL

· Guest management

· Rogue AP Detection: PASS

· Verify Misc. Additional Wireless Features

•  AutoRF: N/A
•  Channel Selection: PASS
•  Channel Spreading
•  Network Scans
•  Spectrum Analysis: PASS
•  Transmit Power Control: PASS
•  Radio Settings Page

                     § Radio Controls

                     § Channel Planning Report

•  Hidden SSID
•  Band Selection and Band Steering: N/A
•  Disabling Legacy 802.11b Bitrates: N/A
•  Meshed Networking: PASS
•  Wired Clients: N/A
•  Wireless Bridging: N/A
•  QoS: FAIL
•  Power Save: N/A
•  Run Dark: : PASS
•  Accessing AP's through local Web Page: PASS

· Test throughput and response time: PASS

· Test speed and duplex from 10/half through 100/full: PASS

· ICMP/Ping testing: PASS

· Get IP addresses through the Staging Center DHCP server: PASS

· Verify that a client can authenticate to through the Staging Center Active Directory server: PASS

· Verify time can be synchronized through SNTP: N/A

· Verify that the internal Cisco AP’s SNMP configuration is recognized by the Staging Center Cisco LMS: N/A

· Verify that the LAP and AP can be managed through web browser, Telnet, console/serial, SSH: PASS, FAIL, FAIL, PASS

· Verify network security through Meraki hosted authentication server: N/A

· Verify network security externally hosted

• o RADIUS: PASS/FAIL
• o PEAP: PASS
• o 802.1X: PASS/FAIL
• o WPA: PASS
• o WPA\PSK: PASS
• o WPA2/802.11i: PASS

· Ethernet port failure to the LAP test to verify that the LAP AP powers down

· Power testing

• o Wireless power settings testing: N/A
• o AP testing with wall outlet power and POE: PASS

· Wireless 802.11a channel testing: PASS

· Wireless 802.11g channel testing: PASS

· Wireless 802.11b/g channel testing: PASS

· Wireless 802.11n channel-bonding: FAIL

· Mobility Grouping: N/A

· Layer 3 Roaming: Ran out of time to test this

1. In our testing/evaluation of the Meraki RF system several issues have been uncovered that would conflict with Customer's current strategic layer three network architecture. The Meraki AP's although they do communicate to a controller are essentially autonomous access points from a user/data perspective.  The AP's control/statistics information is communicated to the Meraki Cloud Controller (MCC) through a secure tunnel.  User data packets are tagged with the appropriate VLAN # and put on an 802.1q trunk between the AP and an access layer switch.  In Customer's current RF architecture the user and control data would travel through a secure LWAPP tunnel to a controller within the Customer LAN.  When the user data gets to the controller it is assembled into an 802.1Q frame, tagged with the appropriate VLAN and put on a trunk linked to a core on the LAN.  The problem is that in order for the user data from the Meraki AP to get to the core where the RF user VLAN's are defined, it would need to be trunked from the access layer where it is connected up to the core/dist layer.   This is the easiest way to solve the problem but  would essentially mean going back to the architecture that Customer started to migrate away from four years ago.  The current strategic architecture at Customer is a layer three architecture that eliminates general purpose trunked VLANs to the access layer and the spanning tree issues that come with it.  To maintain the layer three architecture and deploy an autonomous RF solution the core based RF user VLANs would need to be defined/created at the access layer.  This method would also be a significant change from the current architecture and require extensive rework of existing sites already migrated to the layer three architecture. 

2. Another issue, due to the AP's being autonomous is that the AP's will need to be added to the RADIUS server.  In Customer's current controller based architecture only the controllers need to be added to the RADIUS server.  This would primarily impact QIP before/during deployment and ongoing maintenance. This also complicates network design and deployment. 

3. The mapping features on the MCC utilize Google Maps that are over 10 years old.

4. When AP's lose contact with the MCC and are then powered down or reset they appear to go offline and clients can't associate/connect.

5. How secure (physically and logically) is the Meraki cloud? This question is far reaching and beyond the scope of this document but should be evaluated before a pilot on a real Customer network.

6. What are Meraki's change control procedures? Would Customer be effected by new code versions that may need to be applied? There where changes made to the MCC during our testing to help allow communications through Customer's proxy servers to the internet from within Customer. Would Customer have it's own controllers and be able to dictate a code blockpoint scheme for Meraki to follow?

7. What is Meraki's support model? What time-zones does Meraki have support staff? Our experience during this evaluation project was that Meraki support was slow to respond to issues, sometimes taking several days. 

8. Are there redundant controllers in the Meraki cloud? How could Customer verify this? Does Meraki have standard failover test procedures? When new code versions are applied, does Meraki do regression testing? 

9. After the change was made to the MCC to enable the AP's to get through the Customer proxy we were no longer able to get to "http://my.meraki.com/".

10. The previous issue and also not being able to get to the AP from a browser session essentially makes the AP unusable. A console port would solve this problem.

11. Would Customer be able to have a dedicated pair of MCC's in different regions?

12. Many MCC features are limited i.e. the firewall feature only allows destination addresses.

13. Documentation is limited and lacks detail.

14. In one test scenario an AP was disconnected from the network and it took several hours for the AP to show as missing on the MCC.

15. RADIUS testing issues: Had started testing RADIUS and after getting things setup properly everything appeared to be working properly. I was able to get an IP address, checked the RADIUS event viewer and it showed the test laptop had successfully authenticated with RADIUS and active directory. I started running Chariot traffic between the wireless laptop and a wired client; the throughput was running at about 60 Mbits. When I started testing the next day I wasn't able to get authenticated. I tried a number of times with no success. Worked with Meraki support for several hours and was able to get authenticated again after powering off (battery and plug) the test laptop. Then tried to authenticate again and failed. Through all this the Meraki support person didn't want to cycle power to reset the Meraki AP. The behavior before the laptop power was cycled indicated that the problem was with the AP. So we decided to cycle power on the Meraki AP. Was able to authenticate numerous times throughout the rest of the RADIUS security testing. Didn't have time to try and re-create this issue, so at this time I'm not really sure what the root cause of the problem was.
 

Following is a list of testing tools and their uses:

1. Chariot
   1. Chariot is essentially an application layer test tool that can generate different types of data streams and files. The test scenarios in this document used a script that opens TCP/IP connections and continuously send packets upto1512 bytes between Chariot end points.
   2. Two key results are to be evaluated; overall Chariot traffic throughput on the network and message response time. 
   3. Chariot test results are to be saved in HTML format and there links found in the test scenario results section.
2. Sniffers
   1. Sniffer traces will be taken at points of high data volume. As the tests are running the sniffer will be left in expert mode to observe error/warnings (i.e. retransmits, TTL expirations, broadcast/multicast storms, etc.) and unexpected traffic (i.e. multicasts traversing an uplink that has no group members attached)
   2. The traces will then be saved for future reference.
3. Aironet      Utilities: Aironet Desktop Utility (ADU) & Aironet Site Survey Utility      (ASSU)
4. Ping      scripts and continuous pings

Following is the basic format of each test scenario:

1. Test Scenario; This is a description/definition of what the test will do and in general how to set it up.
2. Test Objective; this is an overview of what the test is for.
3. Test Equipment; a list of the type and quantity of equipment being used, this will only be listed in the first scenario; the equipment is the same for all tests.
4. Test Setup; is primarily made up of configuration and patch panel allocation files.
5. Test Procedure; how to start the tests, test durations, test cycles, what to show,      save/observe, etc.
6. Test Results; this will either have questions to respond to and/or instruction to save      Chariot, switch/AP log files and Sniffer Traces. All Chariot files are to be saved in      HTML format and pasted into this document.
7. The file naming conventions are as follows

• Chariot file format: “CCScenXTestXDateTime” the date and time are the start date and time of the Chariot test.
• Sniffer traces: CCScenXTestXDateTime, the date and time are the same as the Chariot test.
• Switch log files: genericswitchname.txt, are a log of all show commands done on the switches during testing. 
• Configuration Files: switchname.rtf, are the Cisco switch configurations.