Ghost Autonomy, a company specializing in scalable autonomy software for consumer cars, has secured a $5 million investment from the OpenAI Startup Fund to develop large-scale, multi-modal language models (MLLMs) for autonomous driving. The investment brings Ghost's total funding to $220 million. The new funds will be used to accelerate ongoing research and development of LLM-based complex scene understanding, which is essential for urban autonomy, according to a press release dated Nov. 8.

OpenAI's COO and manager of the OpenAI Startup Fund, Brad Lightcap, said that multi-modal models have the potential to expand the applicability of LLMs to various new use cases, including autonomy and automotive. By combining video, images and sounds, multi-modal models can create a new way of understanding scenes and navigating complex or unusual environments.

Ghost Autonomy's founder and CEO, John Hayes, stated that solving complex urban driving scenarios in a scalable way has long been a challenge for the industry, but LLMs provide a breakthrough that will enable everyday consumer vehicles to reason about and navigate through difficult situations. While LLMs have proven valuable for offline tasks like data labeling and simulation, Ghost is excited to apply these powerful models directly to the driving task to realize their full potential.

Ghost's platform allows leading automakers to bring AI and advanced autonomous driving software into the next generation of vehicles, expanding capabilities and use cases with MLLMs. The company is actively testing these capabilities via its development fleet and partnering with automakers to validate and integrate new large models into the autonomy stack.