Tweasel update #3: Switching to a better unpinning script, fixing bugs and writing docs

Hello again from the tweasel project, where we are creating a web app that analyzes mobile apps for privacy issues. One of the major changes we have made is switching to a different unpinning script for Android, which allows us to bypass certificate pinning more reliably. We have also fixed some bugs that were affecting the installation and usage of our tools and libraries on different platforms and devices. In addition, we have continued working on our documentation and outreach, by creating a new docs site and giving a workshop at the Digitalcourage Aktivcongress. We have also collected some new traffic data that we will use to create and improve our TrackHAR adapters.

Appstraction is an abstraction layer for common instrumentation functions on Android and iOS. It allows you to install, uninstall, start, stop apps and configure their permissions, as well as manage device settings like emulator snapshots, clipboard, proxy, and certificates. Appstraction can also be used for purposes other than mobile privacy.

• Last time, Lorenz already talked about the analysis I had run comparing the unpinning capabilities of objection and httptoolkit/frida-android-unpinning, and we had decided to switch to httptoolkit/frida-android-unpinning. I finally implemented that.

  Doing that, I ran into an annoying incompatibility between JS bundlers: We have the unpinning script as a file in our repository. To inject that using Frida, I had read it using Node’s readFile() function. However, Parcel (which we’re using to produce the production builds) was trying to be too smart. It recognized the script as a JS file and bundled it using our settings. However, its output was not compatible with Frida’s JS engine. Parcel also has two ways for inlining files without bundling them, but those are custom extensions and don’t work with tsx/esbuild (which we use for development). I ended up working around this by renaming the file to .txt, so that Parcel doesn’t recognize it as JS anymore.

• Through the change of the unpinning script, we were also able to fix two issues that were related to how we were calling objection:

  • You can now catch errors in startApp().
  • And we are not leaking the objection process anymore. Through fixing this, I learned about Frida’s eternalize() feature, which lets you keep a script injected in a process even after detaching from it.

• Lorenz made using the ssh capability on iOS a lot easier. Now, you don’t have to specify the IP address of the device anymore as we are automatically starting a usbmuxd proxy. We initially planned to use iproxy for this purpose, but now that we’ve switched from libimobiledevice to pymobiledevice3, it obviously made more sense to use that.

  With this change, Lorenz also implemented support for logging in as mobile instead of root as the root user may not be available depending on the jailbreak. Unfortunately, you do still need to manually install openssh due to a bug in palera1n.

• I had noticed a problem with stuck Frida processes after analyzing a lot of apps on Android: After the phone crashed, there was still a Frida process running and any Frida commands just hung indefinitely. Lorenz implemented a fix for that, which automatically kills such stuck Frida processes ensureDevice().

• We received a bug report that installing appstraction on macOS hung indefinitely and never completed. After a bit of investigation, I found that the installation got stuck at download the Android dependencies using andromatic. We noticed that we actually had an existing issue for this problem that we just forgot about! The problem was that the find-java-home package we were using to try an find an existing Java installation incorrectly returns /usr on newer versions of macOS. To fix this, Lorenz switched to @viperproject/locate-java-home.

• We noticed the automatic installation of Frida on iOS wasn’t working properly. Lorenz fixed this problem by clearing the apt package cache before installing Frida.

• We released these changes in version 1.0.0. That release also contains some of the changes we already previewed in the last update.

• Since then, we have already discovered two new bugs that we’ll have to fix in the next release.

  • We had already required a Node version >= 18 because we are using a bunch of newer language features. However, it turns out that the installation fails no newer versions like Node 20. The problem appears to be that frida-node only provides prebuilt binaries for LTS versions of Node. Looks like we can only support Node 18 for now.
  • We had neglected to test the newer changes on Windows, and—probably unsurprisingly—that resulted in a whole bunch of breakage starting with the postinstall script failing.

Cyanoacrylate is a toolkit for large-scale automated traffic analysis of mobile apps on Android and iOS. It uses mitmproxy to capture the HTTP(S) traffic of apps in HAR format and appstraction to instrument physical devices, or emulators for Android. Cyanoacrylate handles the management of certificate authorities and WireGuard mitmproxy setup automatically. It is designed to analyze the tracking behavior of mobile apps.

• I noticed that my previous implementation of exposing mitmproxy events was incomplete. For example, we were missing the hostname in certificate errors. As such, I significantly extended the existing implementation. We are now exposing pretty much all the information that mitmproxy provides in its events, complete with proper type definitions and docstrings.
• This change was released in version 1.0.0. That release also inherits all changes from the appstraction 1.0.0 release.

Tweasel CLI is a command-line tool that allows you to instrument and analyze mobile apps and their traffic using the tweasel project libraries. You can record the traffic of an Android or iOS app in HAR format (based on cyanoacrylate), and detect tracking data transmissions from the traffic (based on TrackHAR). Tweasel CLI provides a convenient wrapper around these libraries for common use cases, so you don’t have to write any code.

• I made analysing already installed apps easier in CLI. The app ID/path argument is now optional. If you don’t provide it, you will be prompted to select an app from a list of all apps installed on the device. This especially convenient on iOS, where you can’t easily find out the bundle ID of an app.

• Lorenz fixed a bug where we were waiting for the Android emulator before we had even started it.

• These changes are released in version 1.0.0, which also inherits all changes from appstraction 1.0.0 and cyanoacrylate 1.0.0.

• I wrote ghtivity, a small utility for displaying the activity in GitHub repositories in a certain time frame to make writing these devlogs easier. I was previously spending way too much time finding all the changes that happened since the last update and that seemed ridiculous.

  

  • Mere minutes after the first release, I noticed two bugs and had to push a quick v1.0.1 release. :)

• We put some thought into how we want to structure our documenation. Our research documentation regarding trackers (to explain why the TrackHAR adapters are the way they are) will live in a separate section on trackers.tweasel.org. For the API reference docs, we will switch to Typedoc proper.

• And for all other documentation, I have started docs.tweasel.org. So far, we already have an introduction that explains the different parts of our project, two tutorials for installing our tools and libraries and how do do traffic analysis with CLI, and a section describing common problems and how to solve them. We plan to extend this in the future, for example with a background section that explains the technical details of how our tools work and our dependencies.

  • We have an open issue for updating our READMEs.

• Lorenz and I were at the Digitalcourage Aktivcongress in Remscheid, an information exchange and networking events for people active in the data protection/digital rights movement that was organised by Digitalcourage. We gave a workshop on how to install and use our tools. As all workshops at the event were fairly ad-hoc, we don’t have a presentation that we can share, but we essentially just followed the tutorials on our new docs site.

• I want to work on creating new adapters for TrackHAR and properly documenting the existing ones next but noticed that the most recent traffic data we have is now almost a year old. Thus, I ran a new collection on ~1000 apps each on Android and iOS. On Android, I used the monkey to generate some random user input and hopefully trigger more requests. I collected 126,161 requests in total, which should give us a good basis for future work.
  We’ll release the data through our public request database, which is almost ready to be published.