OpenAI has significantly revamped its AI agent 'Codex,' adding features such as 'PC operation,' 'direct control via browser,' and 'UI mockups through image generation.'



OpenAI has released an update to its coding assistance AI, Codex. The update includes various enhancements, such as the ability to automatically control a PC, the addition of an in-app browser that allows users to directly give instructions to the AI on web pages, and the addition of image generation.

Codex for (almost) everything | OpenAI

https://openai.com/index/codex-for-almost-everything/




◆PC operation becomes possible
This update allows Codex to recognize all apps on your PC using its own cursor, enabling you to click and input commands. For example, you can prompt a tic-tac-toe app with 'Run this app in Xcode, play it, and fix any bugs.'



Codex automatically checked the current status and recognized that the app's project was open in Xcode.



The Codex uses its own cursor to click the run button.



The game has started, so Codex will begin a test play. Codex clicks on the top left square.



Then, a bug was discovered where 'the computer moves two moves forward every time the player clicks to make a move.'



The code will be automatically corrected, and the tic-tac-toe game will start again.



As originally intended, Codex automatically processed everything to the point where the tic-tac-toe game could be played normally.



The proprietary cursor used by Codex is independent of other Codex instances or user cursors, and it is said that there are no problems running multiple agents in parallel.

Please note that this update is only available for macOS users.

◆Direct instructions can now be given from the in-app browser.
This update enables native integration between the Codex and the web. The Codex now includes an in-app browser, allowing you to directly specify changes to specific parts of a website. For example, below, you can specify the tagline and instruct it to 'make the font smaller and shorten the text.'



The Codex handles the processing automatically.



This update allows you to issue commands to web applications running on your localhost. The plan is to extend this in the future so that the Codex can control the entire browser.

◆Image generation using GPT Image 1.5 is now possible.
The Codex now includes an image generation function. For example, you can specify a webpage and instruct it to 'generate and embed an image at the top.'



The image was generated using GPT Image 1.5 and embedded within the page as instructed.



Images can now be generated within the same workflow when creating product concepts, mockups, and other materials.

◆ Over 90 types of plugins released simultaneously
In addition to skills and apps, over 90 plugins combining AI agents with MCP servers that bridge the gap between external systems were released simultaneously. This gives Codex more ways to take action.

Installing the plugin is extremely easy and only requires a few clicks.



You can use the plugin by specifying it with '@' in the prompt.



◆Support for the entire software development process
Features such as GitHub integration, the ability to run multiple terminals, and connecting to remote development environments via SSH have been added.

The instructions were to 'group the most recent issues posted to this project by topic and compile them into a spreadsheet.'



Codex retrieves data from GitHub, automatically classifies topics, and organizes them for you.



◆ Schedule function
This update allows you to reuse existing conversation threads. At the same time, a scheduling feature has been added to the Codex, enabling you to continue long-term tasks that span several days or weeks.

An example prompt is provided: 'Check Slack, Gmail, Google Calendar, and Notion and let me know what seems important.'



When prompted with 'Keep monitoring,' a scheduled task was added to run every hour.



This update will be rolled out gradually to Codex desktop app users who are signed in with ChatGPT.

Related Posts:

in AI,   Software,   Web Service, Posted by log1d_ts