Google announces “Project Mariner” that can automatically operate Chrome with AI – GIGAZINE

On December 11, 2024 local time, Google announced an AI service that can understand and automatically operate information on the browser.Project Mariner” was announced. Using Project Mariner, you can automatically perform complex operations such as “Searching and summarizing the email addresses of each company based on the company names compiled in a spreadsheet.”

Project Mariner – Google DeepMind
https://deepmind.google/technologies/project-mariner/

Google introduces Gemini 2.0: A new AI model for the agentic era
https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/

Project Mariner is an AI assistant that can perform complex operations according to user instructions. Users can simply give instructions in natural language, such as “Execute ○○ based on the information ○○ in this spread.” When the user gives instructions, the instructions and Chrome screenshots are sent to Gemini on the cloud, and cursor operations, searches, form input, etc. are automatically executed based on Gemini’s analysis results.

The video below is a demo of Project Mariner. Simply enter the task you want Project Mariner to perform, and the AI ​​will automatically understand and analyze the task and carry out the task in sequence.

Project Mariner | Solving complex tasks with an AI agent in the Chrome browser [full length] – YouTube

In the video, with Google Sheets open, the video says, “Memorize this list of companies. Then, find the company’s website and find the email address to contact them. Remember this for later use.” We are presenting instructions to Project Mariner to “deploy it.”

Then, Project Mariner started taking screenshots.

Automatically search the web for these companies’ sites.

Once you find the company’s website you are looking for, we will look for the email address for inquiries on the website.

Repeat this for each company in your list.

Once the series of tasks is completed, the email address for inquiries will be displayed for each company.

In addition, Project Mariner was announced on December 11, 2024.Gemini 2.0”, which utilizes Gemini 2.0’s advanced natural language understanding and reasoning capabilities to interpret both incoming and spoken requests.

Google engineer Adi Osmani says, “A user can simply ask, “Find a job near me,” and Project Mariner will understand that request, navigate to the relevant job search site, and match the user’s location. Customize your search based on your preferences.”

Google says Project Mariner will test AI agent performance on real-world web tasksWeb Voyagerachieved a high score of 83.5%. Regarding this result, Google said, “While the execution of tasks by AI is not necessarily accurate or fast, it shows that it is becoming technically possible for an AI agent to perform tasks within the browser.” says.

Project Mariner takes security seriously and is restricted to only working within the active tab so that users know what Project Mariner is doing, as well as certain sensitive Ask the user for final confirmation before performing a sensitive action. In addition, actions that may directly impact the user’s rights or property, such as “entering credit card number or billing information”, “accepting website cookies”, and “agreeing to terms of use”, are restricted.

Additionally, Project Mariner has learned to prioritize instructions from the user, even in the event of third-party prompt injection attacks, making it difficult to follow malicious instructions from external sources. . This makes users less susceptible to fraud and phishing, even if malicious instructions are hidden in emails, documents, or websites.

According to Google, at the time of writing the article, Project Mariner was being tested by reliable testers. A waiting list for becoming a tester is also available.

Project Mariner Trusted Tester Waitlist
https://docs.google.com/forms/d/e/1FAIpQLSe2J4BvD48E-57giEiXIDz_yZeqGmX0Q3AvvR_LfzpRat2kGQ/viewform