Data is a key force driving AI adoption, providing a constant source of innovation. According to recent research, 96% of CxOs recognize that generative AI is critical to the success of any organization, and 70% of an employee’s tasks have the potential to be automated by gen AI and other technology. However, the main barrier that really stands in the way of enterprise adoption of AI is data accessibility.
According to the Google Cloud Data and AI Trends Report 2024, 66% of organizations report at least half of data is dark, and much of this data is unstructured and unmanaged today. As a data leader in your organization, you will be a key driver of data, from reaching new customers, innovating processes and products, training teams, and so much more. Google Cloud can help connect all your business data to AI, providing teams with efficient, high-powered data insights.
In 2023 and 2024, Google Cloud significantly improves the functional integration of data plus AI:
- In 2023, Google Cloud introduced over 350 new data analytics features, marking a 60% increase from the previous year. This included new gen AI capabilities to accelerate the work of data teams, and AI infused into the data platform through integration of BigQuery with Vertex AI and Gemini models.
- In 2024, Google Cloud integrated more data + AI capabilities into BigQuery.
Google Cloud even announced that in the second half of 2024, more new innovations and mature generation AI features will be released to fully support your data + AI journey, whether on Google Cloud Platform or Google Workspace.
These included:
- Gemini in BigQuery: Deliver AI-powered experiences for data engineering, data exploration and analysis, governance, and security tasks. This includes Gemini capabilities in BigQuery, such as code generation, completion and interpretation (SQL, Python), data insight, data preparation, and partitioning and clustering recommendations.
- Gemini in Looker: Feature Creation offers AI-driven formula assistance that enables users to explore data and create metrics from complex formula and presentation generation, giving a new way to send your analysis directly to your presentation, and finally uses Gemini to deliver engaging documents that integrate seamlessly with Google Workspace. Users of all types can simply chat with your BigQuery data and Looker models for instant and actionable insights.
- BigQuery’s unified data platform: now empowers you to analyze unstructured, structured, and open-format data seamlessly using Vertex AI, Spark and SQL integrations. Google Cloud’s support for Iceberg open format extends to optimizing your data for price-performance automatically . Additionally, Google Cloud announced the general availability of the Delta format, expanding the list of open data formats naively supported.
- Support for open-source Apache Kafka and Apache Spark data streaming and processing: enabling you to harness server-less Spark for rapid, versatile, and scalable data processing on your platform, while utilizing Apache Kafka to gain real-time insights and act on them instantly.
- Real-time streaming from Analytics Hub:Through Analytics Hub’s enhanced data publisher capabilities and Pub/Sub streams, you can subscribe and access real-time data sources within Analytics Hub, manage access to shared topics at scale, helping to ensure data security and compliance. In addition, the integration of Analytics Hub with Google Marketplace means easy commercialization, with AI and data marketplaces making it possible to buy and sell business assets such as datasets, models, and streams to support faster decision making using the latest information.
- A data migration customer offer and enhanced data migration services tooling can help you reduce migration risk, provide upfront payments to cover system costs and export fees, and provide funds to cover migration service costs. Additionally, Google Cloud’s migration services tools speed time to value for data workloads and help you save on migration and export costs.
“Google Cloud continues to strengthen its AI-ready data ecosystem. Gemini integration is an example of the gen AI augmentation we’re seeing that will drive innovation and enhance use cases for data teams and information workers. Platform unification, like the innovations we’re seeing with BigQuery, will make things simpler and easier for customers looking at data platform migrations.” – Doug Henschen, VP & Principal Analyst, Constellation Research, Inc.
Table of Contents
Table of Contents
Gemini data agents for improved productivity
Gemini in BigQuery offers AI-driven capabilities for data preparation, exploration, analysis, governance, and security throughout your data journey. It also provides intelligent recommendations to boost user productivity and optimize costs.
Following the updates at Next ‘24, Google Cloud introduced new preview features such as data insights and data preparation. Additionally, previously announced features like SQL and Python code assistance, data canvas, and partitioning and clustering recommendations are now generally available.
Generating SQL code in BigQuery using Gemini
Wunderkind, a global performance marketing solution, utilizes the data canvas feature in Gemini within BigQuery for investigation and exploration, simplifying query visibility and conserving time and resources for the data team.
“For any sort of investigation or exploratory exercise you know will result in multiple queries, there really is no replacement. It’s saved us so much time and mental capacity” – Scott Schaen, VP of Analytics, Wunderkind
Using Gemini in Looker to generate slides
With Gemini in Looker, features like formula assistant and slide generation(now available in preview) allow information workers to interact directly with their data. You can now create calculated fields instantly without complex formulas. Automatic slide generation can help produce compelling presentations with insightful text summaries of your data.
Conversational Analytics is revolutionizing how organizations engage with their data. Picture simply chatting with your data to receive instant, actionable insights—this transforms the role of analysts by freeing them from constant report generation and empowering business users with true self-service capabilities.
This method goes beyond basic question-and-chart interactions by using advanced language models to navigate through data, provide summaries and uncover automated insights, ensuring important information is not overlooked.
Google Cloud has been engaging to incorporate Conversational Analytics throughout the entire portfolio. Looker users will be able to launch a separate chat client and have more flexibility to transition to Looker Studio for fine-tuning dashboards and reports.
Ease of use with a unified, AI-ready data platform
BigQuery can help prepare all your data for AI. At Next ’24, Google Cloud announced BigQuery will be the unified platform for your data-to-AI journey, designed to be multimodal, multi-engine, and multicloud.
To assist, Google Cloud created a direct integration between BigQuery and Vertex AI, allowing you to access AI models and fine-tune LLMs with your enterprise data, which enhances model consistency and accuracy. New query features in BigQuery are also added using vector indexing, enabling AI to work with your data directly at its source. BigQuery now supports Google’s ScaNN algorithm for efficient batch vector searches. Additionally, Google Cloud had incorporated the latest Gemini models into BigQuery and included new safety and grounding enhancements.
“Switching to BigQuery has transformed our ability to access, understand, and use data at Veo. With its direct integration into other Google Cloud solutions, we get greater accessibility and faster insights, unlocking significant impact and enablement across our organization. Even non-technical team members feel comfortable using BigQuery to run ad-hoc analyses themselves, freeing up time for our analytics team to work on high-value projects.” – Max Schuman, Senior Data Scientist, Veo
An increasing number of customers are using BigQuery to handle unstructured data, such as images, audio, and video, through object tables, with usage surging over 600% year-over-year. BigLake, BigQuery’s open storage engine, enables the user to analyze multimodal and open structured data on single platform with a fully managed Apache Iceberg experience, to create a streaming, fully managed and AI-optimized open lakehouse. At Next ‘24, Google cloud introduced key enhancements for BigLake to support an open, multiformat, and multimodal platform, including a unified runtime metastore that works across various engines and open-format tables like Parquet and Apache Iceberg. Extra supports were added for more open formats, including Delta Lake, which is now generally available. Additionally, new workflow and scheduling features in BigQuery, now generally available, offer data teams increased automation for their data pipelines.
Effortless data processing and streaming
Apache Spark has gained popularity as a data-processing engine, particularly for data engineering tasks, with server-less Apache Spark usage on Google Cloud increasing by over 500% in the past year. BigQuery’s newly integrated Spark engine allows you to process data using PySpark in the same way you use SQL, and like the rest of BigQuery, it is fully server-less, eliminating the need to manage compute infrastructure.
Managing Apache Kafka traditionally involved handling multiple clusters across various cloud providers and on-premises setups. Many customers have expressed a desire for a simpler solution. Now, you can easily set up an Apache Kafka cluster in any of your projects using Google Cloud Managed Service for Apache Kafka, directly from the Google Cloud console. This managed service automates Kafka operations and security, enabling scalable streaming analytics and integration with operational systems.
Streaming data is crucial for industries needing to share real-time information with partners and customers. For example, a retailer might want to provide real-time inventory updates to Consumer Packaged Goods (CPG) companies for improved fulfillment visibility. To facilitate real-time data sharing and monetization from BigQuery, Google cloud is previewing Pub/Sub topic sharing in Analytics Hub. Pub/Sub is a globally recognized messaging service for reliable, large-scale data streaming. Analytics Hub, BigQuery’s data exchange platform, allows thousands of companies to securely share hundreds of petabytes weekly, without data duplication. This new feature supports the curated sharing of streaming data, centralized access management, and real-time discovery of valuable data from other organizations.
Speed up your transition to the cloud for AI readiness
Migrating your data to the cloud is one of the most effective ways to prepare for the AI era. To help organizations speed up their data and AI initiatives, Google Cloud is launching a data platform migration incentive program for data warehouses and data lakes, along with improved BigQuery migration tools. It’s now simpler to move all types of data and workloads—whether multimodal data, SQL, Spark, or Python—to Google Cloud. Start your journey to the Data Cloud today by taking the data strategy assessment and reaching out to our team to fast-track your innovation with AI-driven enterprise data.
Microfusion have being using AI technology to achieve business benefits for enterprise data. If you are interested, please contact Microfusion for consulting. We look forward to continuing to bring the most popular and front-line topics to you.
This article is translated and adapted from the official Google Cloud blog.
