Hot off the Wires 🔥

We're eavesdropping on the smartest minds in research. 🤫 Don't miss out on what they're cooking up! In this section, we dissect some of the juiciest tech research that holds the key to what's next in tech.⚡

Do you notice something peculiar in these two images?

If you spotted that the ratings are overlaid with the hotel name in the first image and that the quantity popup went off the screen in the second image (preventing users from proceeding to buy stuff), then bingo!

These are glaring examples of non-crash UI bugs in Android apps.

Although these screenshots date back to 2018, many of us experience similar bugs while navigating through the interface of some or the other apps. These embarrassing (and sometimes frustrating) bugs highlight a critical, yet often dreaded, aspect of software development: Testing.

Most developers acknowledge the need for extensive testing but find it absolutely mundane & tend to prioritize other tasks.

Thankfully, the world of AI is offering a helping hand. Tools like GitHub Copilot have made it a breeze to automate writing unit tests, etc. for your codebase.

But what about the automation of UI testing? Enter VisionDroid, a tidy, new vision-driven approach to automated UI testing using the power of multimodal LLMs!

Hold on a second... VisionDroid? A vision-driven approach to debugging? Is it just us, or does this sound eerily familiar to our beloved newsletter, "The Vision, Debugged"? 🤔 Looks like great minds think alike – or debug alike!😂

Forging the Fundamentals

All jokes aside, VisionDroid is seriously cool stuff. But before we dive into VisionDroid's magic, let's break down some key terms:

So, what’s new?

Now, you might be thinking, "Don't we already have automated testing?" Well, yes, but there's a catch. Most current automated UI testing techniques are great at catching “crash bugs” – those frustrating moments when an app suddenly closes. But they often miss subtler, non-crash bugs that can be just as annoying for users.

These sneaky bugs might cause misaligned buttons, overlapping text, or a sequence of interactions across multiple screens that doesn't actually crash the app. Spotting these issues typically requires human testers, which is time-consuming and doesn't scale well as apps grow more complex.

This is where VisionDroid shines. Developed by a team of innovative researchers, it uses the power of multimodal large language models (MLLMs) – specifically, GPT-4 – to detect non-crash, functional bugs in mobile apps.

Under the hood…

Now onto understanding how VisionDroid tackles the challenge of identifying non-crash bugs.

Text-Image Alignment

This first step is like giving the AI a detailed map of the app's interface

• Basic information about app, the various GUI pages & information about the multiple widgets on each GUI page are extracted from the manifest file & the view hierarchy file of the app

• Screenshots of the GUI pages are captured & the various widgets are annotated using bounding boxes, having colors based on the widget’s functionality

• Most of this is done using a Python wrapper of UIAutomator

• This alignment ensures that GPT-4 has a clear understanding of the GUI’s layout and elements

Function-Aware Explorer

Here's where things get really cool…

• GPT-4 is prompted to automatically explore various functionalities by using the aligned text & visual information to construct prompts to identify the functionality being tested & the next potential action (like tapping, swiping, and interacting with different features)

• This logical sequence of GUI page screenshots, along with the textual descriptions obtained during exploration are stored in a testing history database

• During this exploration, VisionDroid also employs a few-shot prompt to identify intra-page non-crash bugs within the individual screenshots (if you’re curious, then the examples of intra-page bugs for the examples in the few-shot prompt are obtained from a ‘bug example database’)

Logic-Aware Bug Detector

Now for the really clever part. Finding bugs that span across multiple screens (inter-page bugs) requires a more sophisticated approach, so here’s what VisionDroid does…

• The sequence of screenshots in the testing history database is converted into a graph:

  • Nodes represent screenshots, i.e., the UI page being explored

  • Edges signify transitions between pages

  • Edge weights are assigned based on the semantic similarity of function names, generated by the exploration prompt in function-aware explorer

• The Louvain algorithm is applied to detect communities within this graph, highlighting logical sub-sequences of related screenshots

• These sub-sequences are then analyzed by GPT-4 using another few-shot prompt to spot inter-page bugs

Feel free to dive into their code to understand more about these various processes.

Why does this matter?

VisionDroid significantly outperforms existing deep learning-based UI issue detection techniques and automated GUI testing tools.

When taken for a spin in the wild, VisionDroid was able to identify 83 non-crash bugs across 105 popular apps from Google Play Store (including Expensify, Sygic, DigiCal, etc.). Even more impressive, 37 of these were completely new issues that had slipped past human testers and other automated tools till date!

Developers of respective apps have already fixed 10 of these bugs and confirmed 9 more, proving VisionDroid's real-world value.

Imagine the potential impact on companies like Uber, Airbnb, or mobile banking apps. These businesses rely heavily on their mobile interfaces, and even small bugs can lead to frustrated users and lost revenue.

VisionDroid could be a game-changer in these high-stakes environments, catching issues before they ever reach users, potentially saving millions in support costs and lost business.

Spark & Trouble are rooting for this amazing innovation to make its way into the list of most used test-automation frameworks very soon. Who knows? Maybe one day, "testing" will no longer be a dreaded word in the developer's vocabulary! 😉

Key Takeaways

(screenshot this!)

10x Your Workflow with AI 📈

Work smarter, not harder! In this section, you’ll find prompt templates 📜 & bleeding-edge AI tools ⚙️ to free up your time.

Fresh Prompt Alert!🚨

Feeling like a couch potato with dreams of becoming a gym rat? 🥔💪 We've got just the ticket!

This week's Fresh Prompt Alert is your personal fitness guru in a box. Whether you're aiming to sculpt abs of steel or just want to touch your toes without pulling a muscle, this prompt's got your back (and your core, and your glutes).

So, grab your sweatbands and protein shakes, because it's time to turn those fitness fantasies into sweaty realities! 👇

3 AI Tools You JUST Can't Miss 🤩

• 📃 Alva AI - Your trusted co-pilot system for daily tasks management

• 📈 Bitscale - Empower growth teams to research, personalize and generate content at scale

• 📝 InspNote - AI-powered tool to capture spontaneous thoughts and ideas

Spark 'n' Trouble Shenanigans 😜

“When life gives you lemons🍋, make … buildings🏢”

Sounds out of place? 🤔 Well, with AI, today almost anything is possible! Check out these jaw-dropping AI-powered building designs created by architectural designer Fatih Ekși using food as inspiration (biomimetic architecture on a whole new level), that’s definitely creating waves on Instagram! 👇

Fatih hasn’t revealed the exact tools that he has used to come up with these mind-bending concepts. What’s your guess? Reply to us with your guesses 😉

This is an incredible example of how AI can be used as a powerful tool for creatives, enabling design beyond what was possible till today! Simply bewildering 🤯