Sniffers are the tools that allow you to intercept, analyze and change all requests that pass through them. They are helpful when you need to extract information from a stream or create a desired server response. They can test a product with a backend, a front-end, and different teams with their versioning.
In this article, I will talk about the main sniffing functions that can be useful to QA. I will try not to go into theory but to focus on practice. The most popular representatives of traffic analyzers are now WhireShark, Fiddler, and Charles Proxy. Here I’ll talk about the Charles Proxy since I actively use it myself.
What is Charles Proxy
Charles Web Debugging Proxy is a monitoring tool for HTTP and HTTPS traffic. It acts like a proxy server (intermediate link) between the application under test and the server on the backend, allowing not only to see but also to intercept and edit requests.
The main advantage of Charles Proxy and sniffers, in general, is the ability to view traffic, including from mobile devices, which significantly helps the work of a tester of client-server mobile applications.
Initial setup
Testing the mobile application
Launch Charles Proxy on a computer located in the same local subnet as the mobile device with the application under test.
Typically, the connection is configured over Wi-Fi. In the Wi-Fi settings of the mobile device, you must specify the IP address of the computer and the standard port of the tool 8888 as a proxy server (the password remains blank).
The IP address of the computer can be found through the command line (ipconfig) or in the Charles Proxy itself (Help -> Local IP Address).
The same address can be found in the connection instructions available in Help -> SSL Proxying -> Install Charles Root Certificate on mobile device remote browser.
After saving the settings, Charles Proxy will be able to read the HTTP traffic of the mobile device. To watch decrypted HTTPS traffic, additional manipulations are required — you need to install the Charles Proxy SSL certificate in the browser on the mobile device.
You can download the certificate at: chls.pro/ssl (the address where the certificate is downloaded can also be found in the instructions Help -> SSL Proxying -> Install Charles Root Certificate on mobile device remote browser). Further, in iOS, it must be made trusted (in Settings -> General -> Profiles).
On Android, installed certificates are verified in Settings -> Trusted Credentials on the User tab.
The main difference in the settings for this OS is that with Android 6.0 and higher, a special configuration must be added to Androidmanifest to allow viewing protected traffic. In production, this configuration is removed for security reasons.
Testing on a PC
In this case, additional certificates must be installed on the PC. To download and install, you need a link from Help -> SSL Proxying -> Install Charles Root Certificate.
The certificate is installed in trusted root folders.
Interface
Charles Proxyis’ interface is simple. On the left is a list of intercepted requests; on the right — details.
There are two main tabs in the list of requests — Structure and Sequence.
In the first case, requests are sorted by host-folders. Hovering over any of them, you can get all the information about the number of requests to this root host, the share of successful ones, timings, sizes, etc. In fact, the same information can be obtained from the developer panel in a browser. By selecting a specific URL, you can see the response code, protocol versions, content, etc. The request body, headers, cookies (if any) can be viewed in different formats — even in HEX.
You can use the query context menu to configure locks, retry, and modify queries.
On the Sequence tab, queries are displayed by time in a custom table. When the request started, you can see how long it lasted, its size, status, etc. Hovering over a specific line, we get the same information about the body, headers, etc.
If there are too many requests on the screen, you can clear them or stop interception. There you can also turn throttling on and off (I’ll tell you more about these functions later). The basic setting of each of the functions is carried out through the menu. The control panel buttons act as a kind of On / Off toggle switches.
Filtering
Charles Proxy has many options for request filtering.
Let’s start with the Structure tab. The most primitive is to copy the host and paste it into the Filter field. This way, we will only see requests from this host. You can achieve approximately the same result if you select Focus from the context menu of the host. The rest of the requests will be collected in Other Hosts. If at the same time open the Sequence tab and mark the Focus setting, then the list will contain information only about those queries that were selected on the Structure tab.
The Sequence tab has a similar filter.
Charles Proxy knows how to work with regular expressions. To do this, on the Sequence tab, select Settings and check the Filter uses regex item. We enter a simple regular pattern in the search field.
For example, like this
^ \ w {4} \.you can select all queries with four letters at the beginning of the hostname followed by a period.
You also can enable Autoscroll of the query list or specify the maximum number of lines.
In Charles Proxy, you can filter not only the display but also the recording of requests. To do this, go to Proxy -> Record settings and set conditions on one of the tabs — Include or Exclude — so we enable or disable the recording of requests for this host.
A similar result can be achieved using blocklists. You can enable the host in the block list from the context menu (the Blocklist command) or add it to Tools -> Blocklist, where you should check Enable Blocklist.
A blocked request can be interrupted in two ways (the appropriate one must be selected in the settings):
- reset the connection;
- return error 403.
The choice of blocking method affects the response we receive.
You can experiment. Let’s include our test bench in the Blocklist by choosing a simple connection reset. Use the context menu of the request, repeat it (the Repeat command), and get the following response:
The request status is Failed, and the error description indicates that the connection was dropped.
SSL traffic
Earlier, we’ve successfully installed an SSL certificate to view encrypted traffic. Now we have to enable SSL proxying for the desired host in the Charles Proxy itself. This can be done through the context menu of a specific host.
To not enable each host one by one, you can go to Proxy -> SSL Proxying settings and on the first SSL Proxying tab enable Enable SSL Proxying.
Similar to the Include and Exclude tabs, you can add or exclude specific hosts. If the lists on these tabs are not filled, we will read traffic from all hosts by default.
Breakpoints
Probably one of the most popular features of Charles Proxy allows you to intercept requests.
You can install Breakpoint from the context menu of the request. After that, all similar requests will be intercepted. They can be viewed and edited.
To check how this works, you can use a repeat request (Repeat from the same context menu).
You can change everything — from the header to the authorization token. When editing is finished, you can select Execute, and a repeated request will appear in Charles Proxy, which will be sent to the server, and then returned with a response. At this moment, it will be possible to view and edit the response that the application will receive — the Edit response field will appear.
Editing a request, you can enter deliberately incorrect data and see how the server responds. You can also edit the response (adding incorrect data) and use it to test the front. You can leave the correct data but change the code — see how the front perceives the information transmitted through the API.
Map remote
Another popular Charles Proxy feature is server response spoofing. So we can replace the response of one host with the response of another. This is configured through Tools -> Map Remote.
Notice that there are Import and Export buttons in the lower-left corner. They allow you to exchange settings — transfer them from one workplace to another.
For example, we can update the contours. I will send a request to the dev-contour, but I want to get a response from the test bench. To do this, create a new item in the Map Remote Settings list. Map From — where the request was originally sent; Map to — where we get the response.
This function can be helpful if we caught an error in production, for example, when loading a table, and there is not enough information in the dev-contour for full testing. Instead of re-filling information in each contour, we can copy the server response to the desired contour and see how it behaves after fixing the error.
Map Local
The main difference between Map Local and the previous function is that it is replaced not with the response of another server but with the contents of a local file. The settings look the same, but instead of the second server, we specify the local path to the file with the response.
Rewrite
This function can be helpful if you need to rewrite the data that is sent to Charles Proxy. Unlike simple editing, Rewrite allows you to set change rules and work in automatic mode. You can change and add headers, search and replace text in the body of a request or response. You can even change the response status.
Rewrite is convenient when we don’t want to set a breakpoint every time and change requests manually. This way, we can imitate the work of third-party partners. To be honest, Postman implements this feature in a more user-friendly way. But since we are talking about Charles, we will not deviate from this example.
This can be configured through the Rewrite settings available in the context menu. The only drawback of the tool is that each replacement rule is written separately.
In addition to modifying requests and responses, we can disable caching or cookies (No caching and Block cookies). These options are similar to those of the developer panel in the browser. In both cases, the list of hosts for which the setting is valid is configured. If the list is empty, then caching and cookies are disabled on all intercepted hosts.
Throttling
Charles Proxy helps to test a bad connection by artificially limiting the bandwidth. This feature is most helpful in testing desktop applications because, on mobile devices, the quality of the connection can be controlled through the developer panel.
The function is configured through Proxy -> Throttling settings.
Restrictions can be enabled either for hosts from the list or for all intercepted requests. There are several presets out of the box, but all communication parameters can be manually adjusted. For most situations, the basic settings are ok for me.
Repeat Advanced
While full-fledged load testing is best in dedicated tools, Charles Proxy has one basic setting that helps meet the minimum requirements. The Repeat Advanced function (available through the context menu of an intercepted request) allows you to repeat the same request as many times as needed. After configuration, a separate session will open, where the details of each of the requests will be visible.
Conclusion
Of course, the list of Charles Proxy features doesn’t stop here. There are many useful features — from redirecting the domain name to another IP-address, to automatically saving the received responses.
Note that Charles Proxy is paid. You can use the trial version. Once every 5–7 minutes, a pop-up window with the version will be displayed on top of it, and once every 30 minutes, it will turn off, while sessions are not saved. Decide for yourself if it interferes with your work.
Overall, Charles Proxy is a good tool. I would not say that it is very different from the competitors, but it is more adjusted for MacOS since it came to this platform earlier. Although now it can no longer be said that some kind of sniffer is fundamentally better or worse. They all solve approximately the same problems similarly.
The author of the article: Artem Holevko, Maxilect.
PS. Subscribe to our social networks: Twitter, Telegram, FB to learn about our publications and Maxilect news.
