Over on Twitter, Paul asks how to verify that a native application is using TLS.

For a PC, it’s pretty simple, just run Fiddler and watch the traffic. If you see any HTTP requests (other than those labeled “Tunnel to”, indicating a HTTP tunnel used for HTTPS traffic) from the Process of interest, that traffic is insecure and could be intercepted.

Macs and Mobile Devices

For Mac, iOS, Android, Windows Phone, Windows RT or other devices, the first step is to install Fiddler on a Windows or Linux PC (or Virtual Machine) and configure its proxy to point at the Fiddler instance (e.g. that machine’s IP address, port 8888). For now, don’t add the Fiddler root certificate to the device. Launch the application in question and see whether you see insecure HTTP requests. If not, then look to see whether you see any HTTPS requests. If you see only Tunnel to requests but no HTTPS requests, then the app is using HTTPS securely and isn’t willing to accept just any old certificate (like some insecure apps), only a trusted certificate will be accepted. (If you don’t see any traffic at all, try the default browser to make sure you’ve set up the proxy settings properly).

Using Fiddler’s TextView inspector at the top-right of the debugger, you can examine the CONNECT request (“Tunnel to”) Fiddler captured to see which TLS version the client offered, as well as the list of ciphers and extensions the client supports.

If you’d like to see the plaintext of the HTTPS requests, then install the Fiddler root certificate on the device. If you can now see the decrypted requests, the device has a reasonable HTTPS configuration where HTTPS traffic must be signed by a trusted root certificate.

Certificate Pinning

However, if after trusting the root certificate, you can see HTTPS traffic from the device’s primary browser but not the application in question (you still only see only Tunnel to requests) that implies that the app is using Certificate Pinning, whereby only specific certificates (or certificates that have a specific ancestor certificate in their chain) are accepted. To debug the HTTPS traffic from such an application, you’ll need to jailbreak the device and use a tool like the iOS SSL Kill Switch to thunk the HTTPS APIs to allow any certificate. Certificate Pinning is a good security technique, but it can make your application unusable in certain environments.

The one exception to this heuristic for detecting certificate pinning logic is Chrome on iOS; that app ignores the iOS trusted root store due to limitations in the platform APIs.

Configuration Quality

Beyond the scenarios described above, you should test your browsers’ and servers’ TLS support using the great tools at SSLLabs.com.

You can more exhaustively test a client (by installing a local agent) using this Linux application, and you can read about why validation of HTTPS certificates in non-browser software is considered “the most dangerous code in the world.”

-Eric

By day, I build the Fiddler Web Debugger. I’ve recently started integrating telemetry collection into the application for automated exception reporting and to collect information about the user’s environment to ensure that Fiddler testing environments match real-world usage.

The data is fascinating (and I’ll probably blog more about it later), but one data point in particular jumped out at me:

Early data suggests that nearly a quarter of Fiddler users are using the tool under 32bit Windows. This number is significantly higher than I expected and it’s worrisome. Why?

Because well over half of the exceptions reported via telemetry are System.OutOfMemoryException, and 100% of those are from users running 32bit. Now, given the exception’s name, most users assume this means that the system has “run out of RAM”, but this isn’t really the case. In virtually all instances, this exception would more correctly be named System.InsufficientContiguousAddressSpace – the user has plenty of memory available (both RAM and virtual memory backed by the swap file), but that memory is fragmented such that a large allocation (of, say, 100megabytes) cannot be fit within a single address range of the 2 billion addresses available:

If that 100mb allocation were split into two smaller chunks of 50 megabytes each, it could easily fit into the available address space, and you’d never hit the OutOfMemory exception.

When Fiddler is running in a 64bit process, this error is almost never seen because the address space is so overwhelmingly huge (8 terabytes, 128tb on Win8.1+) that fragmentation isn’t a problem like it is in the 2GB address space.

So, what can we do?

The “simple” approach is to just throw the /3GB switch so that the system assigns up to 3GB of address space to the application, but this requires that the user manually adjust their overall operating system configuration, and it doesn’t buy us very much additional space.

The better approach is to stop using contiguous memory and start using data structures that keep portions of the data in reasonably-sized chunks of only a few megabytes each. This is arguably the “right” fix, but it can be considerably more complex, and it would be a major breaking change for Fiddler because Fiddler exposes HTTP request and response bodies as byte[] requestBodyBytes and byte[] responseBodyBytes. Even if we were to change that, popular APIs like GetResponseBodyAsString would end up making contiguous allocations anyway.

Users can use rules to stream large requests and responses without Fiddler keeping a copy, but this limits a lot of Fiddler’s power—streamed and forgotten traffic can’t be modified and isn’t available for inspection. As a consequence, the “stream and forget” approach is best reserved for bodies over 2 gigabytes (since .NET, even in 64bit, cannot use more than 2gb in a single allocation).

So are we out of luck?

No, not quite.

Before creating the byte arrays, Fiddler has to collect all of the traffic. It does so using a MemoryStream because Fiddler can’t know ahead of time how big the request or response will be.

MemoryStreams are tremendously useful objects, but they’re really just a bit of syntactic sugar around a backing array that resizes itself when you attempt to write more data to it than it can hold. The Write method checks the size of the write operation and if space is needed, it calls a private EnsureCapacity method to grow the backing array. Let’s have a look at this method, trimmed for clarity:

The first thing to notice is the code in the orange box; the MemoryStream has an optimization that says: “If the required capacity is less than double the current capacity, then double the current capacity.” This optimization helps ensure that if you’re writing small amounts of data to the MemoryStream over and over, it doesn’t cause the MemoryStream to reallocate the byte array over and over. Reallocations are especially expensive because the data from the old backing array must be copied to the new backing array upon each allocation.

However, this optimization quickly becomes problematic for 32bit Fiddler. Say we’re reading a large file download of 75 megabytes, reading at 15kb per second. The MemoryStream starts out at 16kb, then quickly grows to 32kb, 64kb, 128kb, 256kb, 512kb, 1mb, 2mb, 4mb, 8mb, 16mb, 32mb, 64mb, and finally ends at 128mb. When the download completes, before Fiddler can generate the 75mb responseBodyBytes array, the MemoryStream has tried to allocate 128 contiguous megabytes, an allocation that is 53 megabytes larger than we truly need. Even if we have a 75 megabyte slot available in memory, we need a 128 megabyte slot for the download to succeed. You’ll also notice that the MemoryStream makes no attempt to catch an OutOfMemoryException error and allocate only what is needed if the double-size allocation fails.

Not good.

Also, have a look at the code in the blue box, which is new for .NET 4.6. Prior to the addition of this code, once more than 1GB of data was written to the MemoryStream, any additional writes to the stream would only grow the MemoryStream to the exact size needed. That was awful for Fiddler performance because every single read from the network would result in reallocating the entire array, from 1.000000gb to 1.000064gb to 1.000128gb to 1.000192gb to 1.000256gb, etc. That performance problem has been fixed, but it introduces a massive overallocation—when a MemoryStream with 1GB of data is reallocated, it automatically jumps to just under 2 gigabytes of data capacity—potentially wasting up to nearly a gigabyte of memory. That’s not awesome, even on 64bit.

So, what to do?

While we can’t fix all of these problems, we can improve the situation over the stock MemoryStream because we know a little bit more about how the caller will be using it than the .NET Framework designers could assume.

We replace MemoryStream with a new class which descends from MemoryStream called PipeReadBuffer. The PipeReadBuffer has an overridden Write method that evaluates the bitness of the process, the size of the current response, and, if available, a “hint” about the final body size. The growth algorithm is enhanced such that:

1. Capacity avoids growing past 85kb as long as possible, to keep the allocation off the large object heap.
2. After that, Capacity grows quickly at small sizes, but growth caps at 16mb (on 32bit) to 64mb (on 64bit) at large sizes.
3. The caller can send the PipeReadBuffer a “Hint” about the expected final capacity based on the Content-Length header, if any.

With these optimizations, the PipeReadBuffer can help ensure that waste is minimized in allocations and the risk of OutOfMemoryExceptions is reduced as much as possible.

Elsewhere in the code, we found great opportunities to tune MemoryStream’s growth. For instance, we have a method called GZipExpand that decompresses data that was compressed using gzip. The API we use unzips into a MemoryStream and previously was subject to the same overallocation problem seen in the network reads. Fortunately, the gzip format includes a hint about the size of the uncompressed data in the footer, so we can use that to set the capacity of our MemoryStream ahead of time:

You can also see that here we take advantage of another optimization—if the stream is sized appropriately, we can avoid the allocation inside the ToArray() call by using the GetBuffer() method instead.

These improvements will appear in the next release of Fiddler.

Further explorations: The Bing team has released a promising new RecyclableMemoryStream (announcement) which might be an even better fit for the PipeReadBuffer, as it can avoid allocations on the Large Object Heap; those allocations are particularly a problem for older versions of .NET that don’t offer any way to compact the LOH.

-Eric Lawrence

Fiddler has long been the tool of choice for developers and testers who are building and verifying APIs exposed over HTTP(S). In this post, we’ll explore the existing features Fiddler offers for API Testing and announce new capabilities we’ve released in Fiddler 2.6/4.6.

Composing API Calls

The Composer tab enables the authoring of arbitrary HTTP(S) requests using any HTTP method, url, headers, and body, and the many Inspectors permit examination of responses of any type, including XML, JSON, BSON, and myriad other formats. Over the last few years, the power of the Composer has grown, and it now offers many conveniences, including a Request History pane:

… to allow simple reuse of existing requests. The powerful Scratchpad tab allows you to easily display a list of requests and execute those requests as needed—a great feature for live demonstrations of your API for tech talks and the like:

As you can see in the screenshot above, the Scratchpad even supports executing requests expressed curl syntax, so you can easily exercise APIs that are documented in terms of the parameters passed to that popular tool.

Capturing API Calls

Of course, Fiddler is best known as a proxy debugger, and its ability to capture traffic from nearly any source means that the fastest way to generate API tests is to simply capture the API traffic from an existing client and modify and augment those tests as desired. You can start with a baseline of the calls your clients presently send, and add new tests to probe for performance, security, error handling, and other problems.

One of the most powerful capabilities Fiddler offers is capture of traffic from almost any device (iOS, Android, Windows, Mac, etc); you can easily exercise your mobile clients’ API endpoints without adding any new software to the device itself!

Resending Requests from the Web Sessions List

After you’ve collected a set of requests in Fiddler, you can easily save them to a Session Archive Zip (.saz) file for convenient reuse at any time. To resend the traffic, just load the SAZ file, select the desired requests, and use the Replay submenu on the Web Sessions list’s context menu to resend the requests. Common choices include:

• Reissue Requests – Resend the selected requests as quickly as possible
• Reissue Sequentially – Resend the selected requests, waiting for a response to each before sending the next
• Reissue and Verify – Resend the selected requests, verifying each response’s status code and body match the original response

Of these options, the last is perhaps the most interesting—you can very easily regression test an API set by simply collecting “good” traffic in a SAZ file and then using that traffic as a baseline against which later verifications will be conducted. The verification performed is rather crude, which is why we’re excited to announce…

Automated API Testing

The new APITest extension to Fiddler greatly enhances its power to validate the behavior of web APIs. Click View > Tabs > APITest to show the APITest tab. The bulk of the tab displays a Test List, a list of API requests and Validators.

Tests can be added to the list by simply dragging and dropping Sessions from Fiddler’s Web Sessions list. You can then use the context menu to specify a Validator and optionally mark it with a Note (comment) and optionally assign it to a group.

Validators

Validators offer a lightweight way to judge the success or failure of a test, and support many of the same operators used by Fiddler’s AutoResponder feature. As seen in the screenshot above:

• EXACT:This is a Test – Ensure the status code matches and response body contains (case-sensitively) the text This is a Test 
• Updated via API – Ensure the status code matches and response body contains (case-insensitively) the text Updated via API
• {CODE} – Only ensure the response status code matches the original status, e.g. 201 for the first test
• {BODY} – (Not pictured above) Ensure the response status code matches and the response body is identical 

In addition to the EXACT: operator prefix, the REGEX: and NOT: prefixes are also supported for text searches:

The HEADER: prefix allows you to ensure that the response bears a specified header containing a given value (e.g. HEADER:Content-Encoding=gzip ensures that the response has a Content-Encoding header showing that the response used GZIP compression).

The SCRIPT: prefix allows you to specify a FiddlerScript function in your CustomRules.js file to use to evaluate the response; the named function is passed both the test Session and the baseline (original) Session and can evaluate any criteria you like to return whether or not the test has passed. Beyond examining the headers and body of the response, you could, for instance, evaluate the values in the Session’s Timers object and fail the test if the response took more than 100 milliseconds.

  // This method is called by “SCRIPT:CheckTest” validators; it
  // returns TRUE if the Test passes, and false otherwise.
  static function CheckTest(oRun: Session, oBase: Session): boolean
  {
    // Simplest possible validator
    if (oBase.responseCode != oRun.responseCode)
    {
        oBase[“api-lastfailreason”] = “Mismatched status code…”;
        return false;
    }

    var timeTotal = 0;
    timeTotal = (oRun.Timers.ServerDoneResponse – oRun.Timers.ClientDoneRequest).TotalMilliseconds;
    if (timeTotal > 100)
    {
        oBase[“api-lastfailreason”] = “Performance Too slow…”;
        return false;
    }

    return true;   
  }

Script Events

Before your Test List is run, the BeforeTestList method (if present in your FiddlerScript) is run, permitting you to adjust the requests as needed:

  static function BeforeTestList(arrSess: Session[]): boolean
  {
    // In this method, you can do any setup you need for the test,
    // e.g. adding an Authorization: token OAUTH value that you
    // obtained programmatically…
    var sOAUTHToken = obtainToken();
    if (String.IsNullOrEmpty(sOAUTHToken)) return false;

    for (var i: int=0; i<arrSess.Length; i++)
    {
      arrSess[i].oRequest[“Authorization”] =  sOAUTHToken;
    }
   
    MessageBox.Show(“Token Set. Running ” + arrSess.Length.ToString() +
                    ” tests.”, “BeforeTestList”);

    return true; // Test should proceed; return false to cancel
  }

After all of the individual test cases are executed, the AfterTestList method allows you to validate any post-conditions, log the results of the Test list, or perform other operations. The method is passed a TestListResults object which contains an enumerable list of TestResult objects. Each result contains the baseline (original) Session, the test Session, and an IsPassing boolean indicating whether the test passed. The WriteFailuresToSAZ convenience method will write all failing TestResults to a SAZ file for later analysis.

  static function AfterTestList(listResults: TestListResults)
  {
    for (var oResult in listResults)
    {
      FiddlerApplication.Log.LogString(oResult);
    }

    listResults.WriteFailuresToSAZ(“C:\\temp\\lastfailure.saz”);
  }

Working with Test Lists

A Test List is simply a set of Sessions each of which contains several custom Session Flags (api-testitem, api-Validator, api-LastResult, and api-LastFailReason). As a consequence, these lists can be saved and loaded as regular SAZ files; their “Test List” functionality only lights up when loaded into the APITest tab using its context menu. The menu commands are largely self-explanatory:

• Run Selected Tests runs only those tests that are currently selected in the UI
• Rerun Failed Tests runs all tests that are marked as failing
• Set Comment… sets the Notes column for the selected tests
• Set Validator… assigns the validation criteria for the selected tests
• Set Group… assigns the tests to a UI group, useful for organizing your test list
• Inspect baseline… opens the original Session in Inspectors for viewing or editing
• Promote moves the selected test earlier in the run order
• Demote moves the selected test later in the run order
• Clone creates a copy of the current test, useful when you need to use multiple validators
• Save Test List saves the test list to a SAZ file
• Load Test List adds the tests from a SAZ file to the current test list

You can temporarily disable one or more tests from running by pressing the spacebar with the desired tests selected.

FiddlerCore

While the new APITest extension is offers powerful functionality, many larger enterprises instead choose to use the FiddlerCore .NET Class Library to build their API Testing infrastructure. FiddlerCore allows you to fully customize the behavior of your testing logic, driving the test using the .NET Language of your choice (typically C#). Because FiddlerCore does not utilize any of Fiddler’s UI, it can easily be integrated into existing test automation suites and can be used in console and service applications.

Load Testing APIs

The Fiddler-native Session Archive Zip file format is supported by Telerik Test Studio’s Load Testing product, allowing you to simply import a SAZ file that exercises your API set, configure the number of virtual users and run time, and generate load instantly or on the schedule you set. If you’d like to apply validation logic while your server is under load, run the Load Test in Test Studio and in parallel execute your Test List in Fiddler.

We hope you find Fiddler’s API Testing support useful, and we look forward to your feedback as we continue to enhance the tool.

Eric Lawrence
Telerik

Telerik Fiddler version 4.6 (and v2.6 targeting .NET2) is now available for download. The new version includes several new features and dozens of tweaks and bugfixes, described in this article.

View > Tabs Menu

The new View > Tabs menu offers a list of tabs that are hidden by default.

The Preferences command displays a tab that allows you to edit Fiddler’s Name/Value preferences.

The new APITest command displays the new Fiddler APITest tab that enables easy testing of web APIs.

The AutoSave command (formerly located in the Tools menu) permits you to automatically save Session Archive Zip files on a regular schedule.

Note: In the future, this submenu will become an extension point to allow developers to easily expose optional UI tabs.

Lint Filters

Fiddler’s Lint feature enables you to control how Fiddler reports violations of the HTTP protocol and other errors. Controlled on the Tools > Fiddler Options > General tab, you can leave the If protocol violations are observed setting at the basic “Warn on Critical errors” section or adjust it to “Warn on all errors”; the latter setting performs more tests on traffic to find mistakes that could cause clients or servers to misbehave. However, this setting can get pretty noisy. To allow you to suppress specific warnings, you can now use the Filter link on the Protocol Violation Report dialog.

Each Lint warning now has a unique code consisting of a letter and three digits. The letter prefix indicates the severity:

• L – Common “problem”, low impact, unlikely to break anything.
• M – Significant problem, likely to break functionality in one client.
• H – Important problem likely to significantly break functionality in multiple clients

You can exclude an entire class of warnings by simply including the prefix in the list of exclusions.

Single Session Timeline

The Timeline tab in Fiddler allows you to visualize the download of multiple Sessions at once to permit you to view parallelism and connection reuse. In Fiddler 4.6, the view of a single Session has been enhanced to permit you to visualize how the content of the Session was downloaded. You can determine, for instance, whether the headers were flushed immediately or the client was forced to wait for headers until the body was ready, and you can determine whether download speed improved or regressed as the download proceeded. For instance, in this chart, the server took 2.5 seconds to return the headers and began to slowly stream the body in small chunks. As the download progressed, the speed improved until completion at just over 9.2 seconds:

Interface Tweaks

A new Copy as Image command on each Inspector’s tab context menu allows you to copy an image of the tab’s contents to your clipboard for easy pasting in an email or blog post. The Headers, XML, and JSON Inspectors allow you to easily highlight nodes of interest—simply press the spacebar on a node to temporarily render it with a yellow highlight:

The new Math context menu appears when you right-click on a numeric column in the Web Sessions list; the menu currently offers a single command: Sum and Average, which shows these attributes of the selected Sessions:

The Composer tab now offers a splitter between the headers and body boxes so you can adjust their height as desired:

Pressing the / key while in the Web Sessions list now enters QuickSearch mode in the QuickExec box, selecting any Sessions whose URL contains the text you type.

Hash Support

Fiddler now offers enhanced support for computing hashes of blocks of bytes or strings.

The Tools > TextWizard feature now offers a quick way to get a hash (MD5 to SHA512) of a block text, in either base64 or dashed hexadecimal format:

New methods are also available for FiddlerScript to compute hashes. For instance, you can copy the following block to just inside your Rules > Customize Rules > Handlers class:

 
public BindUITab("Resource Integrity Hashes", "<nowrap><nolink>")
static function ShowSRI(arrSess: Session[]):String
{
var oSB: System.Text.StringBuilder = new System.Text.StringBuilder();
for (var i:int = 0; i<arrSess.Length; i++)
{
if (arrSess[i].HTTPMethodIs("CONNECT")) continue;
        if (!arrSess[i].bHasResponse)
{
oSB.AppendFormat("\r\n// Skipping incomplete response '{0}'\r\n",
arrSess[i].fullUrl);
continue;
}
if (arrSess[i].responseCode != 200)
{
oSB.AppendFormat("\r\n// Skipping non-HTTP/200 response '{0}'\r\n",
arrSess[i].fullUrl);
continue;
}
var sType: String = arrSess[i].oResponse.MIMEType.ToLower();
var bIsScript = sType.Contains("script");
var bIsCSS = sType.Contains("css");
if (!bIsScript && !bIsCSS)
{
oSB.AppendFormat("\r\n// Skipping non-CSS/JS response '{0}'\r\n", arrSess[i].fullUrl);
continue;
}
var sIntegrity = "sha256-" + arrSess[i].GetResponseBodyHashAsBase64("sha256").Replace("-", "")
+"\n\tsha384-" + arrSess[i].GetResponseBodyHashAsBase64("sha384").Replace("-", "")
+"\n\tsha512-" + arrSess[i].GetResponseBodyHashAsBase64("sha512").Replace("-", "");
        if (bIsScript)
{
oSB.AppendFormat('\r\n<script src="{0}"\r\n\tintegrity="{1}"></script>\r\n',
arrSess[i].fullUrl, sIntegrity);
}
else
{
oSB.AppendFormat('\r\n<link rel="stylesheet"\r\n\thref="{0}"\r\n\tintegrity="{1}">\r\n',
arrSess[i].fullUrl, sIntegrity);
}
}
return oSB.ToString();
}


When you save the script, Fiddler adds a new tab to display the Subresource Integrity attributes for the selected response bodies:

FiddlerScript Improvements

BindUITab Enhancements

As seen in the Resource Integrity Hashes example above, the BindUITab attribute allows you to create new tabs inside Fiddler that are populated based on the selected Sessions. BindUITab now offers a second parameter that allows you to specify one or more of the following options:

• <nowrap> – The RichEdit control should not wordwrap lines
• <nolink> – The RichEdit control should not detect or underline urls
• <html> – Instead of using a RichEdit control, the function’s response will be rendered as HTML inside a Web Browser Control

BeforeFiddlerShutdown event

Fiddler now exposes a  BeforeFiddlerShutdown event that enables extensions or FiddlerScript to block shutdown of Fiddler; this may be useful if you wish to prompt the user for permission to lose unsaved work, etc.

A Fiddler extension should attach an event handler:

    FiddlerApplication.BeforeFiddlerShutdown += (o, c) =>
{
c.Cancel = (DialogResult.Cancel ==
MessageBox.Show(“Allow Fiddler to close????”,
“Go Bye-bye?”, MessageBoxButtons.OKCancel));
};

Within FiddlerScript, you can add a method to the existing Handlers class:

    static function OnBeforeShutdown(): Boolean {
return ((0 == FiddlerApplication.UI.lvSessions.TotalItemCount()) ||
(DialogResult.Yes == MessageBox.Show(“Allow Fiddler to exit?”,
“Go Bye-bye?”, MessageBoxButtons.YesNo, MessageBoxIcon.Question,
MessageBoxDefaultButton.Button2)));
}

AutoResponder Improvements

The AutoResponder now supports the NOT: operator inside the METHOD:, HEADER:, and FLAG:, operators. For instance, if you’d like the AutoResponder only to impact requests from Google Chrome, add this rule to the top of your list:

Microsoft Edge Support

Fiddler has been updated to recognize Microsoft’s new Edge browser as a Web Browser, so features like the Process Filter in the Status Bar:

…and in the Browse command in the Fiddler toolbar:

work as expected.

Note: You should use the WinConfig button at the left of Fiddler’s toolbar to enable Windows 10 “Store” applications to run Fiddler. By default, you shouldn’t need to use the WinConfig button for Edge, because Edge’s about:flags enables access to loopback by default.

ImageView Enhancements

FavIcon Preview

When a site uses an .ICO file as its favicon, the icon may contain multiple different images that are used depending on the user’s device and the context in which the icon is rendered. Fiddler’s ImageView Inspector now renders all of the images contained within the .ICO file like so:

JPEG Thumbnails

Websites should generally strip embedded thumbnails from JPEG files. Embedded thumbnails are a common source of bloat, wasteful bytes that aren’t used by the client. However, some sites fail to optimize their images by removing thumbnails. In some cases, an image thumbnail may even contain data which was never meant to be made public, if, for instance, the larger image was cropped without regeneration of the thumbnail. The ImageView now allows you to extract the thumbnail image as a new Session that is added to the Web Sessions list:

AutoSave Enhancements

Fiddler’s AutoSave feature (now found under View > Tabs > AutoSave) now supports several preferences to control its behavior.

Set fiddler.extensions.AutoSave.AlwaysOn to true to have Fiddler automatically activate AutoSave mode when it starts. Set fiddler.extensions.AutoSave.Minutes to the number of minutes to collect traffic between each save operation; the default is 5. Set fiddler.extensions.AutoSave.HeadersOnly to true if you’d like the SAZ file to contain only the request and response headers, omitting the bodies. Set fiddler.extensions.AutoSave.Path to the folder path under which auto-saved SAZ files should be stored.

Additional Upgrade Notifications

By default, Fiddler’s automatic update notifications will only show if a significant change in version number occurs. For instance, say you’re running version 4.6.0.2 and version 4.6.0.3 becomes available. By default, Fiddler will only tell you about this minor update if you manually check for new versions by clicking Help > Check for Updates. When Fiddler 4.6.1.0 becomes available, the updater detects this larger change in version number and prompts you to upgrade.

If you would prefer Fiddler to notify you of every update automatically, use the black QuickExec box below Fiddler’s Web Sessions list to enter the following command:

    prefs set fiddler.updater.BleedingEdge true

With this preference set, you’ll see more frequent notice of upgrades. On one hand, that’s great—you’ll get the latest Fiddler improvements ahead of most other people. On the other hand, as a bleeding edge user, you’re also more likely to uncover any bugs we inadvertently introduce in new versions.

Fiddler Improvement Program

You may now opt-in to sending telemetry information about your PC environment and Fiddler usage. Within your first few boots of Fiddler, you’ll see the following dialog:

Telemetry data is reported over HTTPS and its usage is governed by Telerik’s Privacy Policy. If you later change your mind, you can control your participation using the checkbox Participate in the Fiddler Improvement Program on the Tools > Fiddler Options > General tab. Note: If an administrator has set the BlockUpdateCheck registry key in the HKLM registry hive, users cannot opt-in to the Fiddler Improvement Program.

The Telerik Analytics integration into Fiddler has already yielded several bugfixes and has helped us prioritize our investments in performance and feature improvements. We’ll write more about what we’ve learned from Fiddler Telemetry in a future blog post.

Performance Improvements

One early finding from our Fiddler Telemetry is that a surprising number of users are running on 32bit Windows, where the address space limitations mean that they often run into “out of memory” errors.  To help mitigate this, we’ve made some under-the-hood changes to how Fiddler allocates memory, and this is only the beginning of a larger project to improve Fiddler’s overall performance for everyone.

As a part of our performance investigations, two new commands were added to the QuickExec box, !memory and !threads. Invoke these commands to add information to the Log tab for troubleshooting purposes. If you find Fiddler is running more slowly than expected, sending the output of these commands to us will help narrow down the problem.

We hope you enjoy the new version of Fiddler!

-Eric Lawrence

Fiddler and FiddlerCore offer three different choices for generating interception certificates:

• MakeCert
• CertEnroll
• Bouncy Castle

If you’re so inclined, you can even write your own certificate generator (say, by wrapping OpenSSL) and expose it to Fiddler using the ICertificateProvider3 interface.

On Windows, Fiddler includes the MakeCert and CertEnroll certificate generators by default; you can download the Bouncy Castle Certificate Generator if you like. In contrast, when Fiddler is running on Linux and Mac, only the Bouncy Castle certificate generator is available, and it is included by default.

If you’re using Windows, however, you may wonder which Certificate Generator you should use in Fiddler or for your applications based on FiddlerCore.

In general, I recommend the Bouncy Castle generator, as it has better performance than the default MakeCert generator and it offers more configuration choices than the CertEnroll generator. Another advantage of the Bouncy Castle certificate generator is that the only certificate that (typically) goes in the Windows Certificate store is the root certificate. The server (end-entity) certificates generated for each website are kept in memory and discarded when Fiddler exits; because the Bouncy Castle generator reuses a single private key for all certificates by default, the performance impact of this behavior is minimal.

The only downside to the Bouncy Castle generator is its size: it is ~200KB when compressed, which is 25% larger than FiddlerCore itself.

The CertEnroll generator was added to Fiddler relatively recently; it offers better performance and standards-compliance than the legacy MakeCert generator but it is available only on Windows 7 and later. You can easily switch Fiddler to use CertEnroll inside Tools > Fiddler Options > HTTPS.

The MakeCert generator is the original certificate generator used by Fiddler and it remains the default on Windows today (mostly) for legacy compatibility reasons. It suffers from a number of shortcomings, including the fact that the certificates it generates are not compatible with iOS and (some) Android devices. It generates certificates with a 1024 bit RSA key (which may soon trigger warnings in some browsers) and each certificate has a unique key (meaning that each new secure site you visit triggers the somewhat costly key generation code).

Both the CertEnroll and MakeCert-based certificate generators must store all server certificates in the Windows Certificate store which some users may find confusing:

The storage of (potentially thousands of) server certificates in the user profile can also cause some problems for corporate users who have roaming user profiles, as these certificates are roamed to each workstation as the user logs in. To mitigate that, the Clear server certs on exit checkbox can be set inside the Tools > Fiddler Options > HTTPS > Certificate Provider dialog, or via:

    FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.CleanupServerCertsOnExit", true);

… however, the downside of doing that is that Fiddler must then re-create the server certificates every time it starts. This performance penalty is smaller when using CertEnroll, which reuses a single 2048-bit RSA key, than for MakeCert, which generates unique 1024-bit RSA keys for each site.

FiddlerCore Considerations

To determine which Certificate Generator is in use, be sure to attach the following event handlers:

Fiddler.FiddlerApplication.OnNotification +=
  delegate(object sender, NotificationEventArgs oNEA) { Console.WriteLine(“** NotifyUser: ” + oNEA.NotifyString); };
Fiddler.FiddlerApplication.Log.OnLogString +=
  delegate(object sender, LogEventArgs oLEA) { Console.WriteLine(“** LogString: ” + oLEA.LogString); };

You can then view information about the Certificate Generator in the console when it loads.

Developers building applications atop FiddlerCore should keep the following in mind when deciding which Certificate Generator to use:

MakeCert

• MakeCert.exe is a Microsoft Visual Studio 2008 redistributable file, meaning that you’re licensed to redistribute it if you have an appropriate license to that version of Visual Studio. Microsoft may offer MakeCert.exe as a redistributable in other circumstances, but licensing is provided by Microsoft, not Telerik.
• To use MakeCert.exe, you must include it adjacent to your application’s .exe file.
• MakeCert-generated certificates are not compatible with iOS and some Android devices.
• MakeCert-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

CertEnroll

• The CertEnroll API is available on Windows 7 and later.
• Use CertEnroll by either omitting makecert.exe from the application’s folder or by explicitly setting the preference:

    FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.PreferCertEnroll", true);

• CertEnroll-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

Bouncy Castle

• Bouncy Castle is an open-source PKI and crypto library distributed under the MIT license.

• To use Bouncy Castle, you must include CertMaker.dll and BCMakeCert.dll adjacent to your application’s .exe file.
• Bouncy Castle does not store certificates in the Windows Certificate Store (yay!) but this also means that your application needs to keep track of its root certificate and private key (unless you recreate and retrust it every time the application runs).

  Two preferences are used to hold the key and certificate, fiddler.certmaker.bc.key and fiddler.certmaker.bc.cert. After you first call createRootCert, you should retrieve these preferences using FiddlerApplication.Prefs.GetStringPref and store them somewhere within your application’s settings (registry, XML, etc); the private key should be considered sensitive data and protected as such.  When your application next runs, it should detect whether the key and certificate have already been created, and if so, they should be provided to the certificate generator using FiddlerApplication.Prefs.SetStringPref before any certificates are requested, lest you inadvertently create a new root certificate.

  Rick Strahl wrote a great blog post on this process, including some sample code.

-Eric

Regular readers of my blog know how excited I am about Google’s new open compression engine, Brotli. Support is in Firefox 44 Nightly already and is expected for other browsers. Because Brotli is used inside the WOFF2 font format, many clients already have the compression code and just need to expose it in a new way.

Yesterday’s Fiddler release supports Brotli. To expose Brotli inside Fiddler 4.6.0.5+, download and place brotli.exe inside Fiddler’s %ProgramFiles(x86)%\Fiddler2\Tools\ subfolder and restart to see it appear on the Transformer tab:

After installation, Content-Encoding: br is also supported by Fiddler’s Decode toolbar option, as well as the encoding Infobar:

Test servers are starting to come online for Brotli, particularly now that Google has released a new Brotli module for the nginx web server. For now, you can try downloading this image:

…that is served with brotli encoding. If your browser supports brotli, it will render a star; if not, the image will appear broken unless you set the Decode option in Fiddler’s toolbar so Fiddler will decode the image before the browser sees it.

-Eric Lawrence

You can easily use Fiddler to evaluate what algorithms a client is using to connect to a HTTPS server in Fiddler.

First, adjust Fiddler’s configuration using Tools > Fiddler Options to enable capture of CONNECT tunnels but disable decryption:

Disabling decryption is necessary because Fiddler decrypts traffic using a HTTPS man-in-the-middle technique, which means that when it’s enabled you’ll see what the client and server are using to talk to Fiddler, which could be different than what they’d use if Fiddler were not in the middle.

After making this change, simply load a HTTPS site inside your browser, and double-click on any of the CONNECT tunnel entries shown in Fiddler:

Fiddler’s TextView Request Inspector will show you a parsed view of the Client’s HTTPS handshake:

… and the TextView Response Inspector below will show the parameters chosen by the server for the connection:

In this case, you can see that the Server agreed to a TLS/1.2 connection using RSA for key exchange and 128-bit AES as the symmetric encryption algorithm. The server ignored the ALPN extension indicating that the connection will not be using HTTP2 or SPDY for data transfers.

Fiddler will show you exactly what your client and server agreed upon. If you’re interested in exploring what other clients with other options might negotiate, the SSLLabs Server Test is a great tool. Similarly, if you’re curious about the capabilities of your browser, check out the SSLLabs Client Test.

-Eric

TLDR? – Get the newest Fiddler here. We’re performing a staged rollout of this build; it won’t be on autoupdate until next week.

Under the Hood

As mentioned in our notes about the Fiddler 4.6 release, we’ve started taking a very close look at Fiddler’s performance. Fiddler’s use of the CPU, system memory, and the network have gone under the microscope and this new release includes several major changes to how Fiddler uses threads and memory. If you frequently run Fiddler with a large amount of traffic in parallel, or run Fiddler on a slower or heavily-loaded PC, this new version should provide significantly improved performance. We’ve also improved overall performance by using better algorithms in scenarios like the Find Sessions (CTRL+F) experience.

The !threads and !memory QuickExec commands have been enhanced to provide insights into fine-grained performance details about Fiddler’s operation.

The Performance Tab

The new Performance tab in the Fiddler Options dialog offers choices that can significantly change Fiddler’s runtime performance and memory usage.

The Show Memory panel in status bar controls whether Fiddler’s status bar shows a panel that indicates the current memory usage tracked by the garbage collector. For example, when Fiddler has 64mb of managed memory allocated, the panel looks like this:

Left-click the memory panel to instruct the .NET Framework to perform an immediate garbage collection. Right-click the panel to launch the Fiddler Options dialog box with the Performance tab activated.

The Parse WebSocket Messages checkbox controls whether Fiddler will parse WebSocket streams into individual messages, allowing display in the WebSocket tab and manipulation using the OnWebSocketMessage event handler. Disabling WebSocket Message parsing will reduce CPU usage and may save a significant amount of memory if high-traffic WebSockets are in use. Even if you disable WebSocketMessage parsing globally using this checkbox, it can be reenabled on a per-Session basis by setting the x-Parse-WebSocketMessages flag on the Session object.

The Stream and forget bodies over box controls the maximum size of a message body that Fiddler will retain. By default, the limit is just under 2 gigabytes for 64bit Fiddler and 16 megabytes for 32bit Fiddler; the much smaller default for 32bit helps avoid problems with “Out of Memory” errors when running Fiddler in a small address space. If, while reading a message body, Fiddler finds that it is larger than the threshold, it will configure the body to stream and will “drop” the bytes of the body to conserve memory. If you attempt to inspect a Session which has been dropped, you will see the following notification bar:

Clicking the bar will open the Fiddler Options dialog to allow you to reconfigure the limit for subsequent Sessions.

The If client aborts while streaming dropdown controls Fiddler’s behavior if a response body is streaming to the client but the client closes the connection. Depending on your choice here, Fiddler can continue to read the body from the server (useful if you’re collecting traffic) or abort the Session (useful to save memory and CPU cycles).

The Run Fiddler at AboveNormal Priority alters Fiddler’s default scheduling priority. If you enable this option, Windows will prioritize activation of Fiddler’s threads when they have work to do (e.g. reading a new request or response from the network). You can easily experiment with this option to see whether it improves the overall throughput of your client (browser) and Fiddler.

New QuickFilters

The Session list’s Filter Now context menu has been enhanced with two new filters:

The Hide /1stpath/ filter hides any traffic whose Url path component starts with the specified string.

The Hide Url… option uses the current Session’s Url as the default of a Url filter; you can edit the string to apply more broadly by removing text from the start or end of the string:

High DPI Improvements

Today, only a small number of Fiddler users (< 4%) run Fiddler on Windows systems with a non-default screen DPI, but we want Fiddler to work great for those users too. The latest build of Fiddler includes a number of DPI-related fixes. Fiddler is not yet marked DPI aware in its manifest; if you’d like to see Fiddler in its DPI-aware mode, run Fiddler with the -dpiAware command line argument:

    fiddler.exe -dpiAware

We will continue to make improvements as problems are discovered or reported and expect to eventually set the dpiAware flag by default.

New HTTPS Cipher Option

Fiddler 4 on Windows 7 and later supports modern TLS versions (TLS 1.1 and TLS 1.2) and the HTTPS tab on the Fiddler Options dialog enables you to easily enable these protocols. However, TLS 1.1 and 1.2 remain off-by-default for compatibility reasons.

The Enabled Protocols link on the HTTPS dialog now supports a new token <client>; if present, this token adds to the list of versions offered to the server the latest protocol that has been offered by the client. For instance, with these settings:

… a request from Internet Explorer offering TLS 1.2 will be presented to the server with TLS 1.2 and TLS 1.0 enabled. The advantage of using the <client> token is that if the request fails, many browser clients are configured to “fall back” and attempt negotiation with an earlier protocol version. In this example, if the TLS 1.2 connection fails, the browser will retry with TLS 1.0 and the connection may succeed.

You must include at least one specific TLS version in the HTTPS Protocols list to handle cases where the request was generated by Fiddler itself (e.g. the Composer tab).

Brotli Compression Support

Researchers at Google have developed a new compression algorithm named Brotli that offers significantly better compression than the DEFLATE algorithm used by Gzip. This new compression algorithm is already in use by many browsers today (inside the WOFF2 font format) and it is expected to appear as a HTTP Content-Encoding in early 2016.

Fiddler now supports Brotli as a Content-Encoding everywhere compression is supported; simply download the Authenticode-signed Windows Brotli.exe and place it in the Fiddler2\Tools\ subfolder in your Program Files folder. After you restart Fiddler, you will find a new Brotli option on the Transformer tab and Fiddler APIs like utilDecodeResponse() will be able to decompress Brotli-encoded content:

The new version of Fiddler also has better handling of unsupported compression schemes like SDCH—if a response with Content-Encoding: sdch,gzip is encountered, for instance, the various decoding APIs will decompress with GZIP and then stop without removing the SDCH token.

Hidden Tabs

FiddlerScript’s BindUITab attribute now supports a <hidden> token:

If this token is present, the tab is not shown until the user manually activates it via the View > Tabs menu:

This feature helps you “unclutter” Fiddler by keeping uncommonly-used script tabs hidden.

More Powerful ImageView Extensions

Fiddler’s ImageView Extensions feature allows you to add new commands to the Tools context menu on the ImageView Inspector:

Now you can use a new Options parameter to specify that Fiddler should show the <stdout> or <stderr> results of running the target tool, and a new {out:extension} token enables you to specify that the target tool writes a file that Fiddler should load as a new Session in the Web Sessions list.

For instance, here’s the logic to add a new ToWebP Lossless command to the list:

To use it, add the registry entries and place CWebP.exe in Fiddler’s Tools subfolder. When you invoke the command, Fiddler will run the tool, passing an input temporary file containing the JPEG image to the tool in the {in} parameter and specifying an autogenerated filename with a .webp file extension in the {out:webp} parameter. The cwebp.exe tool will be run, any text from Standard Error will be collected and displayed to the user, and the file named by the {out} token will be reloaded as a new Web Session:

Updated – Show Image Bloat

I’ve also updated the Show Image Bloat add-on (described here) with some additional tweaks and features; the add-on has improved bloat detection for JPEG and GIF files and has other minor improvements. Install the latest build of Show Image Bloat (v2.6) and activate it from the Fiddler Rules menu.

I hope you enjoy these new improvements to Fiddler – Keep sending in your feedback to ensure we’re evolving the tool to best meet your needs.

-Eric Lawrence

I’ve made changes to the latest versions of Fiddler to improve the performance of certificate creation, and to avoid problems with new certificate validation logic coming to Chrome and Firefox. The biggest of the Fiddler changes is that CertEnroll is now the default certificate generator on Windows 7 and later.

Unfortunately, this change can cause problems for users who have previously trusted the Fiddler root certificate; the browser may show an error message like NET::ERR_CERT_AUTHORITY_INVALID or The certificate was not issued by a trusted certificate authority.

Please perform the following steps to recreate the Fiddler root certificate:

1. Click Tools > Fiddler Options.
2. Click the HTTPS tab
3. Uncheck the Decrypt HTTPS traffic checkbox
4. Click the Remove Interception Certificates button
5. Accept all of the prompts that appear (e.g. Do you want to delete these certificates, etc)
6. (Optional) Click the Fiddler.DefaultCertificateProvider link and verify that the dropdown is set to CertEnroll
7. Exit and restart Fiddler
8. Click Tools > Fiddler Options.
9. Click the HTTPS tab
10. Re-check the Decrypt HTTPS traffic checkbox
11. Accept all of the prompts that appear (e.g. Do you want to trust this root certificate)

If you are using Fiddler to capture secure traffic from a mobile device, you will need to remove the old Fiddler root certificate from that device and install the newly-generated Fiddler certificate.

I apologize for the inconvenience, but I believe that the new certificate generator will help ensure smooth debugging with current and future clients.

-Eric Lawrence

Fiddler’s default file format is the SAZ Format, which is just a ZIP file with a particular structure. Unfortunately, sometimes users’ SAZ files get corrupted due to failing disks or incomplete downloads, and when this happens, Fiddler can no longer open them.

Because Fiddler uses a standard ZIP file, surely a good ZIP reader will be able to read some data, right?

Windows Explorer’s primitive ZIP implementation can’t do anything useful:

Alas, not even 7-zip offers any help.

Okay, well, surely you can just use any of the many ZIP Repair tools to extract the data that isn’t corrupt from the file, right?

Alas, a few hour’s worth of research suggests that almost all of the public ZIP repair tools are terrible, unable to handle most forms of corruption. Some claim to work, but the resulting “repaired” archive remains unreadable:

Those tools that seem promising aren’t free, and require spending $30 or so before you can even determine whether they’ll get your data back.

What to do?

Write my own, of course. Most SAZ files are internally quite simple, and it shouldn’t be too hard to recover most data from archives that aren’t encrypted.

Fiddler 4.6.2 will offer a Repair Corrupt option on the dropdown in the Load dialog box:

When you choose this option, Fiddler will enter its archive recovery mode:

Notably, the recovery mode doesn’t especially care whether the recovered ZIP file is a SAZ file. If not, Fiddler will alert you that the file couldn’t be interpreted as a SAZ:

… but the repaired file on your desktop:

… should now be openable by your ZIP reader of choice:

I hope you find this new capability useful, both for Fiddler-generated files as well as any other corrupt ZIP or ZIP-based (e.g. docx, pptx) files you may encounter.

-Eric Lawrence

Sadly, you’re unlikely to get wealthy by writing a book. You should definitely write one anyway.

My Background

People I respect suggest you shouldn’t write (or buy) books on specific technologies, going so far as to say that writing a book was on their top-10 lists of life regrets. Top-10… whoa!

As a consequence, when I was approached to write a book about Internet Explorer in 2009, I turned it down. “No one reads books anymore,” I asserted to the Vice President of the Internet Explorer team. At the time, I was sitting about 6 feet from my bookshelf full of technical books that I’d been buying over the last decade, including a few I’d purchased in the last month.

My counter-factual stance continued for the next few years, even as I served as a technical reviewer for five books on web development and performance. Then, in 2011, as I started pondering the sale of Fiddler, I met some product managers at the proposed acquirer and watched them use Fiddler for a few minutes. I was appalled—these guys had been looking at Fiddler for six months, and seemed to have settled on the most cumbersome and complicated ways to get anything accomplished. It wasn’t really their fault—Fiddler was incredibly feature rich, and I couldn’t fault them for not reading the manual—there wasn’t one. I felt a moral obligation, whether I sold Fiddler or not, to at least write down how to use it.

At the time, my wife was training to run marathons and I had quite a bit of free time in the mornings. Not knowing any better, I did what I assumed most writers do—I took my laptop to the coffee shop in the mornings and started writing. My resolve was aided by two crutches—

1. I was happy to describe Fiddler, feature-by-feature, from top to bottom, left to right (Hemmingway, this wasn’t), and
2. I decided that even if I abandoned the project without finishing a proper book, I could just save the results to a PDF, title it “Fiddler-Manual.pdf” and upload it to the Fiddler website.

I’ll cover the mechanics of actual writing in a future post (surprisingly straightforward, but I have some advice that may save you some time), but for now it suffices to say that after nine months of work a few times a week, I had a book.

My Investment

Writing the first edition took about 110 hours authoring, 20 hours of editing, and 30 hours of designing the cover, fixing formatting, and futzing with the printer/publisher webapp. I spent about $50 on draft copies, $40 or so on the cover photo, $20 on the fiddlerbook.com domain name registration, and about $650 for coffee and pastries consumed while writing. From September 2011 to June 2012, I periodically took a “snapshot” of my progress by printing the work:

Writing took about three months longer than my prediction:

… in part because as I wrote, I discovered what I’ve come to call Book-Driven Development (covered in the next section).

I was proud of the final product, but skeptical that it would earn back even what I spent on coffee.

So, Why Write?

First, it’s something parents and other folks can tangibly hold and appreciate. It’s probably the only way my grandmother will ever have any idea what the heck a “Fiddler Web Debugger” is and why someone might use one.

Second, it’s tangible. Many people have contributed to Fiddler over the years, and I can inscribe a paperback copy and send it to them as a “Thank you.” When the book was finished, I bought a dozen copies and dropped them off in the offices of colleagues who’d made contributions (in the form of bug reports or suggestions) over the years. One of the proudest moments of my life was when I got an email from Mark Russinovich thanking me for the signed copy and noting that it would nicely complement the ebook he’d already bought.

Third, writing a book makes you think very very hard about what you’re writing about, and with a different mindset. The Fiddler book took quite a bit longer to write because I made hundreds of improvements to Fiddler while I was writing the book, because I was writing the book. Almost every time I thought of something interesting to explain, I began to write about it, then realized that whatever it was shouldn’t have been so complicated in the first place. I’d then go fix Fiddler itself to avoid the problem. In other cases, explicitly writing out everything you can do with Fiddler made me recognize some important (and in hindsight, obvious) gaps, and go implement those features. I started calling this process Book-Driven Development and Fiddler was dramatically improved over the authoring of the book. Having said that, this also made writing the book take quite a bit longer—I’d write three or four pages, realize that the feature in question shouldn’t be so complicated, and go implement a checkbox or a button that would do everything the user needed without explanation. Then I’d have to go delete those three or four pages and replace it with “To do <X>, just push the <X> button.”

Fourth, I got to choose what to write about. Fiddler is insanely powerful, but after watching people “in the field” use it, it was plain that most of its functionality is completely unknown to the vast majority of users. While some users watch the videos and read the blog posts, it was clear that there are some number of folks for which a complete book with an end-to-end explanation of the tool is the best way to learn it.

Fifth, it gives you an appreciation for other authors that you may never get otherwise. Marathon runners probably have more respect for other marathon runners than the general public ever will, simply because they know how grueling it is to run 26.2 in a way that someone who hasn’t never will. I think the same is probably true for book-writers.

So, in summary:

1. It’s tangible.
2. You can gift it to contributors.
3. You’re forced to think like a new user.
4. You can drive the direction of usage.
5. You learn to appreciate authors.
6. Self-publishing significantly changes your book’s financial prospects.

Money Matters

One of the challenges with almost any profit-making endeavor is that folks are so coy about the numbers involved. Inspired by John Resig’s post on Programming Book profits, I’m going to share my numbers here. My goal isn’t to brag—I think these are solid numbers, not runaway success numbers, but I want to show why “You’ll never make any money selling a book” is simply untrue.

Having read a bunch of posts like Jeff Atwood’s and Resig’s, I realized that going the traditional publisher route was a bad deal for both the reader and for me– the Fiddler book would have been ~$30 and I’d see maybe two or three dollars of that. Self-publishing is a better deal for the reader (lower prices) and it’s a better deal for me (I get about $6 and $8 respectively). While a traditional publisher would have probably netted me an advance of a few thousand bucks (more than I expected to make) I frankly prefer the “honesty” of being solely responsible for my book’s sales, and the often happy feeling I get when I (obsessively?) check sales stats and find that I sold a handful more copies overnight.

The first edition of Debugging with Fiddler was released in June 2012. The book was self-published on Lulu for the ebook (a PDF) and via CreateSpace (paperback) which was mostly sold on Amazon. The Lulu ebook was sold for a flat $10, while Amazon set the price for the paperback, usually at a small discount off the nominal $18.99 cover price.

Here are the sales figures for the ebook on Lulu:

The paperback sold slightly better, with 2713 English copies sold; the CreateSpace report below includes the 319 copies sold (so far) of the Second Edition:

Beyond the sales of my book, I also agreed to let the book be translated to Korean, Chinese, and Japanese by three local publishers. Each publisher agreed to an advance of about $1500, as well as three or four copies of the translated paperback. Of these, only one publisher (O’Reilly Japan) sends follow-up royalty statements; the book sold approximately 1400 copies in Japanese, and their advance was followed by a royalty check of $1450 in February of 2014.

On March 5th of 2015, I released the Second Edition update, revised to describe the changes in Fiddler over the last three years. This too proved far more successful than I’d ever dreamed. The $14.99 PDF (usually “on sale” for $9.99) took the lion’s share of sales with 840 copies sold, vs. 319 copies of the revised paperback. While the paperback stayed at CreateSpace, I moved to GumRoad for the new ebook for reasons I’ll describe in a future post.

So, how much did I earn? A bit more than $53318 or so thus far– the Euro/GBP exchange rates make the math a bit tricky. I spent about 200 hours of solid work on the First and Second Editions, so this works out to a bit over $250 per hour. Pretty amazing, for a project that yielded so many non-financial rewards!

Results Will Vary

It’s worth mentioning that my sales numbers are almost certainly much higher than they would’ve been “naturally”, but for one critical factor— as the developer of Fiddler, I was in a position to advertise the book both on the Fiddler website and in the Fiddler application itself. Exposed to millions of Fiddler users, this exposure was obviously invaluable and not, alas, something available to most writers.

It’s also the case that as the tool’s author, I benefit from credibility and name recognition (people expect that I’ll know what I’m writing about). As the primary source, I also have the luxury of writing more quickly since I didn’t need to do much research (subject to the “Book driven development” caveats earlier).

My (long overdue) next book project, Blue Badge, is a memoir of my time working at Microsoft, and it won’t benefit from the incredible exposure I had for Debugging with Fiddler. I’m intrigued to see how it sells.

If you’re an aspiring author, or simply interested in book publishing, I hope you found this post useful!

-Eric

My last day at Telerik is December 31st, 2015.

More soon…

TLDR? – Get the newest Fiddler here.

It’s been just over two months since the last significant release, and Fiddler 4.6.2.0 (and v2.6.2.0) are now available.

As always, the latest build includes a slew of bugfixes and minor tweaks, as well as a number of features described in this post.

Default Certificate Generator Changed

Changes coming to certificate validation in browsers and other clients mean that certificates generated by makecert.exe (previously Fiddler’s default generator) will soon be rejected. To address this problem, the default certificate generator on Windows 7 and later has been changed to CertEnroll. (Windows XP and Vista users should consider installing the similar CertMaker Addon).

Unfortunately, if you’re upgrading from an earlier version of Fiddler which used a different certificate generator, you may need to explicitly reset Fiddler’s certificates. Doing so is simple:

1. Click Tools > Fiddler Options.
2. Click the HTTPS tab.
3. Ensure that the text says Certificates generated by CertEnroll engine.
4. Click Actions > Reset Certificates. This may take a minute.
5. Accept all prompts
6. If you are using Fiddler to capture secure traffic from a mobile device or Firefox, you will need to remove the old Fiddler root certificate from that device (or Firefox) and install the newly-generated Fiddler certificate.

If necessary, you can read more about resetting Fiddler’s Certificates or read more about Fiddler’s Certificate Generators.

SAZ Repair

From time to time, users have asked for help with Fiddler Session Archive files (.SAZ or .RAZ files) that are corrupt, either because they are incomplete (e.g. power failed) or they were mangled by an incomplete download or a failing disk drive.

Fiddler 4.6.2 includes a new feature that can recover data from corrupt Session Archive files. If the Session Archive fails to load due to corruption, you’ll be prompted to attempt a repair of the file. Data recovered from the SAZ file will be stored in a new archive and loaded for display.

Notably, this feature may also be useful to recover corrupt .zip, .docx, .xlsx, .pptx, etc files that have nothing to do with Fiddler; give it a try!

FiddlerHook Removed

This release removes the FiddlerHook extension for Firefox. Mozilla is changing their add-on model for Firefox extensions. Short-term, Firefox requires that extensions be signed (and Mozilla has declined to sign FiddlerHook) and over the next year, Mozilla will be removing the XUL Overlay extension model upon which FiddlerHook was based.

Fortunately, you don’t really need FiddlerHook to use Fiddler with Firefox. For HTTP traffic, it will often “just work” and for HTTPS traffic, only minor configuration updates are needed. You can read this post for tips on using Fiddler with Firefox.

Decryption Control

Previously, Fiddler UI only allowed you to exempt certain hosts from HTTPS decryption; if you wanted to only decrypt from a small number of hosts, you were forced to use the script engine. That limitation has been removed via a new option on the Tools > Fiddler Options > HTTPS tab. Simply click the link to toggle between exclusion and inclusion:

Extensibility Improvements

This release adds a number of improvements to Fiddler’s extensibility, from both FiddlerScript and .NET extensions.

BindUIButton

You can now add buttons to Fiddler’s toolbar in a supported way. Simply add a new BindUIButton attribute to a static method in your FiddlerScript file; the string argument is the caption with which to label the button.

For instance:

    BindUIButton(“Copy HAR”)

yields:

Toolbar buttons are added to the left of the toolbar in the order opposite of their listing within the FiddlerScript. Adding images is not supported from FiddlerScript but you can use Unicode Emoji symbols if you’d like.

Fiddler extensions may add to the toolbar using the static method FiddlerToolbar.AddToolstripItem() and may remove entries using .RemoveToolstripItem().

Export to String

In many cases, you may wish to generate a string representing one or more Sessions in either HTTPArchive (HAR) or cURL format. While you can manually use File > Export to generate files of either format, you can now skip the middle-man and export to these types in memory. To do so, simply set the ExportToString option and do not set a Filename option. After the DoExport command completes, the output is found in the OutputAsString option.

For instance, you can add the following to your FiddlerScript:

    BindUIButton(“Copy HAR”)
    ContextAction(“Copy HAR”)
    public static function doCopyHar(arrSess: Session[])
    {
       var oExportOptions = FiddlerObject.createDictionary();
   
        // If you’d prefer to save to a file, set the Filename instead
        //oExportOptions.Add(“Filename”, “C:\\users\\lawrence\\desktop\\out1.har”);

        oExportOptions.Add(“ExportToString”, “true”);
        oExportOptions.Add(“MaxBinaryBodyLength”, 1000000);
        oExportOptions.Add(“MaxTextBodyLength”, 1000000);
   
        FiddlerApplication.DoExport(“HTTPArchive v1.2”, arrSess, oExportOptions, null);
        var sOutput: String = oExportOptions[“OutputAsString”];
   
        Utilities.CopyToClipboard(sOutput);                        
        FiddlerApplication.UI.SetStatusText(“Copied Sessions as HAR”);
    }

…and Fiddler will add a toolbar button and context menu command that copies the Selected Sessions to the clipboard in HAR format. (Tip: You can choose Edit > Paste As Sessions in Fiddler to create new Sessions based on HAR text that you’ve copied from browser tools.)

Alternatively, you can add a similar command to copy the Selected Sessions as a set of cURL commands:

    BindUIButton(“Copy cURL”)
    ContextAction(“Copy cURL”)
    public static function doCopyCurl(arrSess: Session[])
    {
       var oExportOptions = FiddlerObject.createDictionary();
   
        // If you’d prefer to save to a file, set the Filename instead
        //oExportOptions.Add(“Filename”, “C:\\users\\lawrence\\desktop\\out1.bat”);

        oExportOptions.Add(“ExportToString”, “true”);
   
        FiddlerApplication.DoExport(“cURL Script”, arrSess, oExportOptions, null);
        var sOutput: String = oExportOptions[“OutputAsString”];
   
        Utilities.CopyToClipboard(sOutput);                        
        FiddlerApplication.UI.SetStatusText(“Copied Sessions as HAR”);
    }

Invoking on the UI Thread

Fiddler processes Sessions on background threads, but you should only ever manipulate Fiddler’s UI using the UI thread. Only a few of Fiddler’s UI calls are thread-safe; if you’re not sure, your script should use the new FiddlerObject.uiInvoke or FiddlerObject.uiInvokeAsync methods to avoid crashing or corrupting the user-interface.

Load Extensions at Runtime

To support some exciting new work from the community, Fiddler now has the ability to load additional Extensions and Inspectors at runtime; this enables building of more complex add-on systems atop Fiddler’s existing system. To use these APIs, invoke any of these four methods from the UI thread:

    FiddlerApplication.oExtensions.InstantiateInspectorsFromPath(string sPathToInspectors)
    FiddlerApplication.oExtensions.InstantiateExtensionsFromPath(string sPathToExtensions)
    FiddlerApplication.oExtensions.InstantiateExtensionsInFile(FileInfo oFile, bool bWriteToLog, bool bRethrowExceptions)
    FiddlerApplication.oExtensions.InstantiateExtensionByType(Type typeExtension, bool bWriteToLog)

Thank You!

Lastly, I’d like to thank everyone for all of your support over the last twelve years, as Fiddler has evolved from a side project to a fully-supported debugging platform used around the world. I’m excited to see where Telerik takes Fiddler next, and while I’ll be keeping plenty busy in my new job, I expect I’ll remain involved in the Fiddler community (updating the book, and haunting the forum) for quite some time.

Wishing you all the best in 2016 and beyond!

-Eric Lawrence

Fiddler’s Transformer tab has long been a simple way to examine the use of HTTP compression of web assets, especially as new compression engines (like Zopfli) and compression formats (like Brotli) arose. However, the one-Session-at-a-time design of the Transformer tab means it is cumbersome to use to evaluate the compressibility of an entire page or series of pages.

Introducing Compressibility

Compressibility is a new Fiddler 4 add-on¹ which allows you to easily find opportunities for compression savings across your entire site. Each resource dropped on the compressibility tab is recompressed using several compression algorithms and formats, and the resulting file sizes are recorded:

You can select multiple resources to see the aggregate savings:

WebP savings are only computed for PNG and JPEG images; Zopfli savings for PNG files are computed by using the PNGDistill tool rather than just using Zopfli directly. Zopfli is usable by all browsers (as it is only a high-efficiency encoder for Deflate) while WebP is supported only by Chrome and Opera. Brotli is available in Chrome and Firefox, but limited to use from HTTPS origins.

To show the Compressibility tab, simply Install the Addon, restart Fiddler, and choose Compressibility from the View > Tabs menu².

The extension also adds ToWebP Lossless and ToWebP Lossy commands to the ImageView Inspector’s context menu:

I hope you find this new addon useful; please send me your feedback so I can enhance it in future updates!

-Eric

¹ Note: Compressibility requires Fiddler 4, because there’s really no good reason to use Fiddler 2 any longer, and Fiddler 4 resolves a number of problems and offers extension developers the ability to utilize newer framework classes.

² If you love Compressibility so much that you want it to be shown in the list of tabs by default, type prefs set extensions.Compressibility.AlwaysOn true in Fiddler’s QuickExec box and hit enter.

Fiddler and FiddlerCore offer three different choices for generating interception certificates:

• MakeCert
• CertEnroll
• Bouncy Castle

If you’re so inclined, you can even write your own certificate generator (say, by wrapping OpenSSL) and expose it to Fiddler using the ICertificateProvider3 interface.

On Windows, Fiddler includes the MakeCert and CertEnroll certificate generators by default; you can download the Bouncy Castle Certificate Generator if you like. In contrast, when Fiddler is running on Linux and Mac, only the Bouncy Castle certificate generator is available, and it is included by default.

If you’re using Windows, however, you may wonder which Certificate Generator you should use in Fiddler or for your applications based on FiddlerCore.

In general, I recommend the Bouncy Castle generator, as it has better performance than the default MakeCert generator and it offers more configuration choices than the CertEnroll generator. Another advantage of the Bouncy Castle certificate generator is that the only certificate that (typically) goes in the Windows Certificate store is the root certificate. The server (end-entity) certificates generated for each website are kept in memory and discarded when Fiddler exits; because the Bouncy Castle generator reuses a single private key for all certificates by default, the performance impact of this behavior is minimal.

The only downside to the Bouncy Castle generator is its size: it is ~200KB when compressed, which is 25% larger than FiddlerCore itself.

The CertEnroll generator was added to Fiddler relatively recently; it offers better performance and standards-compliance than the legacy MakeCert generator but it is available only on Windows 7 and later. You can easily switch Fiddler to use CertEnroll inside Tools > Fiddler Options > HTTPS.

The MakeCert generator is the original certificate generator used by Fiddler and it remains the default on Windows today (mostly) for legacy compatibility reasons. It suffers from a number of shortcomings, including the fact that the certificates it generates are not compatible with iOS and (some) Android devices. It generates certificates with a 1024 bit RSA key (which may soon trigger warnings in some browsers) and each certificate has a unique key (meaning that each new secure site you visit triggers the somewhat costly key generation code).

Both the CertEnroll and MakeCert-based certificate generators must store all server certificates in the Windows Certificate store which some users may find confusing:

The storage of (potentially thousands of) server certificates in the user profile can also cause some problems for corporate users who have roaming user profiles, as these certificates are roamed to each workstation as the user logs in. To mitigate that, the Clear server certs on exit checkbox can be set inside the Tools > Fiddler Options > HTTPS > Certificate Provider dialog, or via:

    FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.CleanupServerCertsOnExit", true);

… however, the downside of doing that is that Fiddler must then re-create the server certificates every time it starts. This performance penalty is smaller when using CertEnroll, which reuses a single 2048-bit RSA key, than for MakeCert, which generates unique 1024-bit RSA keys for each site.

FiddlerCore Considerations

To determine which Certificate Generator is in use, be sure to attach the following event handlers:

Fiddler.FiddlerApplication.OnNotification +=
  delegate(object sender, NotificationEventArgs oNEA) { Console.WriteLine(“** NotifyUser: ” + oNEA.NotifyString); };
Fiddler.FiddlerApplication.Log.OnLogString +=
  delegate(object sender, LogEventArgs oLEA) { Console.WriteLine(“** LogString: ” + oLEA.LogString); };

You can then view information about the Certificate Generator in the console when it loads.

Developers building applications atop FiddlerCore should keep the following in mind when deciding which Certificate Generator to use:

MakeCert

• MakeCert.exe is a Microsoft Visual Studio 2008 redistributable file, meaning that you’re licensed to redistribute it if you have an appropriate license to that version of Visual Studio. Microsoft may offer MakeCert.exe as a redistributable in other circumstances, but licensing is provided by Microsoft, not Telerik.
• To use MakeCert.exe, you must include it adjacent to your application’s .exe file.
• MakeCert-generated certificates are not compatible with iOS and some Android devices.
• MakeCert-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

CertEnroll

• The CertEnroll API is available on Windows 7 and later.
• Use CertEnroll by either omitting makecert.exe from the application’s folder or by explicitly setting the preference:

    FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.PreferCertEnroll", true);

• CertEnroll-generated certificates “pollute” the user’s Certificate Store and you should consider offering a mechanism to clear them.

Bouncy Castle

• Bouncy Castle is an open-source PKI and crypto library distributed under the MIT license.

• To use Bouncy Castle, you must include CertMaker.dll and BCMakeCert.dll adjacent to your application’s .exe file.
• Bouncy Castle does not store certificates in the Windows Certificate Store (yay!) but this also means that your application needs to keep track of its root certificate and private key (unless you recreate and retrust it every time the application runs).

  Two preferences are used to hold the key and certificate, fiddler.certmaker.bc.key and fiddler.certmaker.bc.cert. After you first call createRootCert, you should retrieve these preferences using FiddlerApplication.Prefs.GetStringPref and store them somewhere within your application’s settings (registry, XML, etc); the private key should be considered sensitive data and protected as such.  When your application next runs, it should detect whether the key and certificate have already been created, and if so, they should be provided to the certificate generator using FiddlerApplication.Prefs.SetStringPref before any certificates are requested, lest you inadvertently create a new root certificate.

  Rick Strahl wrote a great blog post on this process, including some sample code.

-Eric

Regular readers of my blog know how excited I am about Google’s new open compression engine, Brotli. Support is in Firefox 44 Nightly already and is expected for other browsers. Because Brotli is used inside the WOFF2 font format, many clients already have the compression code and just need to expose it in a new way.

Yesterday’s Fiddler release supports Brotli. To expose Brotli inside Fiddler 4.6.0.5+, download and place brotli.exe inside Fiddler’s %ProgramFiles(x86)%\Fiddler2\Tools\ subfolder and restart to see it appear on the Transformer tab:

After installation, Content-Encoding: br is also supported by Fiddler’s Decode toolbar option, as well as the encoding Infobar:

Test servers are starting to come online for Brotli, particularly now that Google has released a new Brotli module for the nginx web server. For now, you can try downloading this image:

…that is served with brotli encoding. If your browser supports brotli, it will render a star; if not, the image will appear broken unless you set the Decode option in Fiddler’s toolbar so Fiddler will decode the image before the browser sees it.

-Eric Lawrence

You can easily use Fiddler to evaluate what algorithms a client is using to connect to a HTTPS server in Fiddler.

First, adjust Fiddler’s configuration using Tools > Fiddler Options to enable capture of CONNECT tunnels but disable decryption:

Disabling decryption is necessary because Fiddler decrypts traffic using a HTTPS man-in-the-middle technique, which means that when it’s enabled you’ll see what the client and server are using to talk to Fiddler, which could be different than what they’d use if Fiddler were not in the middle.

After making this change, simply load a HTTPS site inside your browser, and double-click on any of the CONNECT tunnel entries shown in Fiddler:

Fiddler’s TextView Request Inspector will show you a parsed view of the Client’s HTTPS handshake:

… and the TextView Response Inspector below will show the parameters chosen by the server for the connection:

In this case, you can see that the Server agreed to a TLS/1.2 connection using RSA for key exchange and 128-bit AES as the symmetric encryption algorithm. The server ignored the ALPN extension indicating that the connection will not be using HTTP2 or SPDY for data transfers.

Fiddler will show you exactly what your client and server agreed upon. If you’re interested in exploring what other clients with other options might negotiate, the SSLLabs Server Test is a great tool. Similarly, if you’re curious about the capabilities of your browser, check out the SSLLabs Client Test.

-Eric

TLDR? – Get the newest Fiddler here. We’re performing a staged rollout of this build; it won’t be on autoupdate until next week.

Under the Hood

As mentioned in our notes about the Fiddler 4.6 release, we’ve started taking a very close look at Fiddler’s performance. Fiddler’s use of the CPU, system memory, and the network have gone under the microscope and this new release includes several major changes to how Fiddler uses threads and memory. If you frequently run Fiddler with a large amount of traffic in parallel, or run Fiddler on a slower or heavily-loaded PC, this new version should provide significantly improved performance. We’ve also improved overall performance by using better algorithms in scenarios like the Find Sessions (CTRL+F) experience.

The !threads and !memory QuickExec commands have been enhanced to provide insights into fine-grained performance details about Fiddler’s operation.

The Performance Tab

The new Performance tab in the Fiddler Options dialog offers choices that can significantly change Fiddler’s runtime performance and memory usage.

The Show Memory panel in status bar controls whether Fiddler’s status bar shows a panel that indicates the current memory usage tracked by the garbage collector. For example, when Fiddler has 64mb of managed memory allocated, the panel looks like this:

Left-click the memory panel to instruct the .NET Framework to perform an immediate garbage collection. Right-click the panel to launch the Fiddler Options dialog box with the Performance tab activated.

The Parse WebSocket Messages checkbox controls whether Fiddler will parse WebSocket streams into individual messages, allowing display in the WebSocket tab and manipulation using the OnWebSocketMessage event handler. Disabling WebSocket Message parsing will reduce CPU usage and may save a significant amount of memory if high-traffic WebSockets are in use. Even if you disable WebSocketMessage parsing globally using this checkbox, it can be reenabled on a per-Session basis by setting the x-Parse-WebSocketMessages flag on the Session object.

The Stream and forget bodies over box controls the maximum size of a message body that Fiddler will retain. By default, the limit is just under 2 gigabytes for 64bit Fiddler and 16 megabytes for 32bit Fiddler; the much smaller default for 32bit helps avoid problems with “Out of Memory” errors when running Fiddler in a small address space. If, while reading a message body, Fiddler finds that it is larger than the threshold, it will configure the body to stream and will “drop” the bytes of the body to conserve memory. If you attempt to inspect a Session which has been dropped, you will see the following notification bar:

Clicking the bar will open the Fiddler Options dialog to allow you to reconfigure the limit for subsequent Sessions.

The If client aborts while streaming dropdown controls Fiddler’s behavior if a response body is streaming to the client but the client closes the connection. Depending on your choice here, Fiddler can continue to read the body from the server (useful if you’re collecting traffic) or abort the Session (useful to save memory and CPU cycles).

The Run Fiddler at AboveNormal Priority alters Fiddler’s default scheduling priority. If you enable this option, Windows will prioritize activation of Fiddler’s threads when they have work to do (e.g. reading a new request or response from the network). You can easily experiment with this option to see whether it improves the overall throughput of your client (browser) and Fiddler.

New QuickFilters

The Session list’s Filter Now context menu has been enhanced with two new filters:

The Hide /1stpath/ filter hides any traffic whose Url path component starts with the specified string.

The Hide Url… option uses the current Session’s Url as the default of a Url filter; you can edit the string to apply more broadly by removing text from the start or end of the string:

High DPI Improvements

Today, only a small number of Fiddler users (< 4%) run Fiddler on Windows systems with a non-default screen DPI, but we want Fiddler to work great for those users too. The latest build of Fiddler includes a number of DPI-related fixes. Fiddler is not yet marked DPI aware in its manifest; if you’d like to see Fiddler in its DPI-aware mode, run Fiddler with the -dpiAware command line argument:

    fiddler.exe -dpiAware

We will continue to make improvements as problems are discovered or reported and expect to eventually set the dpiAware flag by default.

New HTTPS Cipher Option

Fiddler 4 on Windows 7 and later supports modern TLS versions (TLS 1.1 and TLS 1.2) and the HTTPS tab on the Fiddler Options dialog enables you to easily enable these protocols. However, TLS 1.1 and 1.2 remain off-by-default for compatibility reasons.

The Enabled Protocols link on the HTTPS dialog now supports a new token <client>; if present, this token adds to the list of versions offered to the server the latest protocol that has been offered by the client. For instance, with these settings:

… a request from Internet Explorer offering TLS 1.2 will be presented to the server with TLS 1.2 and TLS 1.0 enabled. The advantage of using the <client> token is that if the request fails, many browser clients are configured to “fall back” and attempt negotiation with an earlier protocol version. In this example, if the TLS 1.2 connection fails, the browser will retry with TLS 1.0 and the connection may succeed.

You must include at least one specific TLS version in the HTTPS Protocols list to handle cases where the request was generated by Fiddler itself (e.g. the Composer tab).

Brotli Compression Support

Researchers at Google have developed a new compression algorithm named Brotli that offers significantly better compression than the DEFLATE algorithm used by Gzip. This new compression algorithm is already in use by many browsers today (inside the WOFF2 font format) and it is expected to appear as a HTTP Content-Encoding in early 2016.

Fiddler now supports Brotli as a Content-Encoding everywhere compression is supported; simply download the Authenticode-signed Windows Brotli.exe and place it in the Fiddler2\Tools\ subfolder in your Program Files folder. After you restart Fiddler, you will find a new Brotli option on the Transformer tab and Fiddler APIs like utilDecodeResponse() will be able to decompress Brotli-encoded content:

The new version of Fiddler also has better handling of unsupported compression schemes like SDCH—if a response with Content-Encoding: sdch,gzip is encountered, for instance, the various decoding APIs will decompress with GZIP and then stop without removing the SDCH token.

Hidden Tabs

FiddlerScript’s BindUITab attribute now supports a <hidden> token:

If this token is present, the tab is not shown until the user manually activates it via the View > Tabs menu:

This feature helps you “unclutter” Fiddler by keeping uncommonly-used script tabs hidden.

More Powerful ImageView Extensions

Fiddler’s ImageView Extensions feature allows you to add new commands to the Tools context menu on the ImageView Inspector:

Now you can use a new Options parameter to specify that Fiddler should show the <stdout> or <stderr> results of running the target tool, and a new {out:extension} token enables you to specify that the target tool writes a file that Fiddler should load as a new Session in the Web Sessions list.

For instance, here’s the logic to add a new ToWebP Lossless command to the list:

To use it, add the registry entries and place CWebP.exe in Fiddler’s Tools subfolder. When you invoke the command, Fiddler will run the tool, passing an input temporary file containing the JPEG image to the tool in the {in} parameter and specifying an autogenerated filename with a .webp file extension in the {out:webp} parameter. The cwebp.exe tool will be run, any text from Standard Error will be collected and displayed to the user, and the file named by the {out} token will be reloaded as a new Web Session:

Updated – Show Image Bloat

I’ve also updated the Show Image Bloat add-on (described here) with some additional tweaks and features; the add-on has improved bloat detection for JPEG and GIF files and has other minor improvements. Install the latest build of Show Image Bloat (v2.6) and activate it from the Fiddler Rules menu.

I hope you enjoy these new improvements to Fiddler – Keep sending in your feedback to ensure we’re evolving the tool to best meet your needs.

-Eric Lawrence

I’ve made changes to the latest versions of Fiddler to improve the performance of certificate creation, and to avoid problems with new certificate validation logic coming to Chrome and Firefox. The biggest of the Fiddler changes is that CertEnroll is now the default certificate generator on Windows 7 and later.

Unfortunately, this change can cause problems for users who have previously trusted the Fiddler root certificate; the browser may show an error message like NET::ERR_CERT_AUTHORITY_INVALID or The certificate was not issued by a trusted certificate authority.

Please perform the following steps to recreate the Fiddler root certificate:

Fiddler 4.6.1.5+

1. Click Tools > Fiddler Options.
2. Click the HTTPS tab.
3. Ensure that the text says Certificates generated by CertEnroll engine.
4. Click Actions > Reset Certificates. This may take a minute.
5. Accept all prompts

Fiddler 4.6.1.4 and earlier

1. Click Tools > Fiddler Options.
2. Click the HTTPS tab
3. Uncheck the Decrypt HTTPS traffic checkbox
4. Click the Remove Interception Certificates button. This may take a minute.
5. Accept all of the prompts that appear (e.g. Do you want to delete these certificates, etc)
6. (Optional) Click the Fiddler.DefaultCertificateProvider link and verify that the dropdown is set to CertEnroll
7. Exit and restart Fiddler
8. Click Tools > Fiddler Options.
9. Click the HTTPS tab
10. Re-check the Decrypt HTTPS traffic checkbox
11. Accept all of the prompts that appear (e.g. Do you want to trust this root certificate)

If you are using Fiddler to capture secure traffic from a mobile device or Firefox, you will need to remove the old Fiddler root certificate from that device (or Firefox) and install the newly-generated Fiddler certificate.

I apologize for the inconvenience, but I believe that the new certificate generator will help ensure smooth debugging with current and future clients.

-Eric Lawrence

Fiddler’s default file format is the SAZ Format, which is just a ZIP file with a particular structure. Unfortunately, sometimes users’ SAZ files get corrupted due to failing disks or incomplete downloads, and when this happens, Fiddler can no longer open them.

Because Fiddler uses a standard ZIP file, surely a good ZIP reader will be able to read some data, right?

Windows Explorer’s primitive ZIP implementation can’t do anything useful:

Alas, not even 7-zip offers any help.

Okay, well, surely you can just use any of the many ZIP Repair tools to extract the data that isn’t corrupt from the file, right?

Alas, a few hour’s worth of research suggests that almost all of the public ZIP repair tools are terrible, unable to handle most forms of corruption. Some claim to work, but the resulting “repaired” archive remains unreadable:

Those tools that seem promising aren’t free, and require spending $30 or so before you can even determine whether they’ll get your data back.

What to do?

Write my own, of course. Most SAZ files are internally quite simple, and it shouldn’t be too hard to recover most data from archives that aren’t encrypted.

Fiddler 4.6.2 will offer a Repair Corrupt option on the dropdown in the Load dialog box:

When you choose this option, Fiddler will enter its archive recovery mode:

Notably, the recovery mode doesn’t especially care whether the recovered ZIP file is a SAZ file. If not, Fiddler will alert you that the file couldn’t be interpreted as a SAZ:

… but the repaired file on your desktop:

… should now be openable by your ZIP reader of choice:

I hope you find this new capability useful, both for Fiddler-generated files as well as any other corrupt ZIP or ZIP-based (e.g. docx, pptx) files you may encounter.

-Eric Lawrence