iPhone's Siri Can Soon Understand How Applications Work With A New Advanced Model

Rohit Kulkarni

Rohit Kulkarni

Apr 15, 2024

For the 90s kid, AI and its first encounter is “Siri”, a AI-powered voice assistant which was launched along with the iPhone 4S’ features in 2011.

iPhone's Siri Can Soon Understand How Applications Work With A New Advanced Model

The Evolution of Siri and Apple’s Gen AI

This is also when one had first-hand experience of iPhone, which many of 90s kid did not necessarily had.

Siri, made life easy, by executing many tasks ranging from setting up a call to setting up an alarm.

It was new and fun and made life a tad easy if not a lot easier.

However, in the recent years there has not been much movement in “Siri”. In the interim, world witnessed AI explosion and many fields and almost all companies are planning to integrate it into their business processes.

With the AI explosion and adoption, it is getting reported that our own “Siri” would also undergo this transformation and enter the AI pipeline, only the come out smarter from the other end.

Apple and its work on Gen AI for Siri has been doing rounds in the technology sector for some time.

The Rise of Ferret-UI and its Impact on Mobile AI

As per a new research paper published by Cornell University, there is a new MLLM (Multimodal Large Language Model) which understands how a phone’s interface works. 

Titled as “Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs”, the paper throws light on how despite coming a long way, the technology still not is fully successful when it comes to user interface interactions with screens.

Launched in October 2023, Ferret UI is an MLLM which is being developed with the aim to understand the UI screens as well as to understand how the apps in phone work. This model, paper claims to be having “referring, grounding, and reasoning capabilities.”

Ferret is an open-source, MLLM that was released between Apple and Cornell University, resulting from extensive research on how large language models could recognise and understand elements within pictures. This implies that any UI which has Ferret running underneath can handle queries like a ChatGPT or Gemini does.

There are diverse aspect ratios and compact visual elements found in smartphone displays, and the same is one of the primary challenging when it comes to enhancing the AI understanding of app screen.

What the model does is, it magnifies the details and leverages the enhanced visual features to understand even the smallest icons as well as buttons. The paper further explains that when it comes to ability to understand and interact with app interfaces, Ferret UI has far surpassed the contemporary models.

This digital assistant, is touted to execute complex tasks within these apps in future. What it would mean in future is that Siri could now undertake complex tasks such as booking a flight or making a reservation since the app would interact with the app.

Rohit Kulkarni
Rohit Kulkarni
  • 897 Posts

Subscribe Now!

Get latest news and views related to startups, tech and business

You Might Also Like

Sep. 8, 2022

Samsung Launches The Wall All-In-One and Flip Pro: Is This The Future Of Display Technology?

Samsung has launched The Wall All-In-One – the modular MicroLED it says is revolutionizing the future of display and the Flip Pro, which is an interactive display. Both were unveiled at the InfoComm India 2022 which is India’s Professional AudioVisual (Pro AV) and Systems Integration Technology Exhibition. This took place in Mumbai from September 5-7. […]

Jul. 28, 2022

Google Street View Launches In India Across These 10 Indians Cities! Plans To Expand To 700,000 Kms, 50 Cities In 2 Years

Google’s Street View is finally available in India a decade after it was prevented from capturing data for its Street View services. Second coming Street view offers a 360-degree interactive panorama feature initially for 10 Indian cities with data from local partners Tech Mahindra and Mumbai-based Genesis International. Its entry into India is facilitated by […]

Jul. 10, 2022

This Electricity-Free Cooler Developed By IIT Researchers Can Replace Air Conditioners! How It Work?

Indian Institute of Technology Guwahati researchers have built a ‘Radiative Cooler’ which does not require electricity to operate. This is an affordable and efficient ‘passive’ radiative cooling system that can serve as an alternative to ACs. The coating material is an electricity-free cooling system that can be applied in the rooftops and functions during both […]

Jun. 27, 2022

This New Battery Will Enable Electric Cars To Run 1000 Kms In Single Charge! Will This Transform Electric Mobility?

Chinese company Contemporary Amperex Technology Co Ltd (CATL) has developed a battery that gives EVs a driving range of over 1,000 km or 621 miles on a single charge. About the company The company is China’s leading automotive lithium-ion battery maker. Tesla, Volkswagen, BMW, and Nio are among the companies for which CATL supplies batteries. […]

Recent Posts

Related Videos


Subscribe Now!

Get latest news and views related to startups, tech and business

who's online