Category: Flutter

Google AI Studio’s Gemini API is the fastest way to start building with Gemini, our next-generation family of multimodal generative AI models.

Gemini API Audio Capabilities:

Gemini can respond to prompts about audio.

• Interprets and explains audio content through descriptions, summaries, and Q&A
• Transcribes entire audio files or specific segments
• Provides targeted answers and transcriptions for designated audio sections”

Gemini API Audio Processing Rules

• Audio is represented at a rate of 25 tokens per second, equivalent to 1,500 tokens per minute.
• Gemini only comprehends English-language speech but can recognize non-speech sounds like birdsong or sirens.
• The maximum audio length for a single prompt is 9.5 hours, with no limit on the number of files, but a combined total not exceeding 9.5 hours.
• Audio files are downsampled to 16 Kbps and multi-channel audio is mixed into a single channel.

Share Audio with Gemini API: Two Easy Options

1. Pre-upload the audio file: before sending your prompt, making it readily available for processing.

2. Embed the audio file: directly within your prompt request as inline data, streamlining the process.

Pre-upload the audio file

• Upload Audio Files with Ease. Use the File API to upload audio files of any size.
• When your total request exceeds 20 MB (including files, text prompts, and instructions), switch to the File API for a seamless upload experience.”

curl -o sample.mp3 https://storage.googleapis.com/generativeai-downloads/data/State_of_the_Union_Address_30_January_1961.mp3

Now, upload that audio file:

audio_file = genai.upload_file(path='sample.mp3')

This code downloads an audio file and then uploads it to the platform using the genai.upload_file(path='sample.mp3')method.

Send prompts to Gemini about the uploaded file

After uploading your audio file, you can send a prompt to Gemini using the generateContent method.

# Initialize a Gemini model appropriate for your use case.
model = genai.GenerativeModel(model_name="gemini-1.5-flash")

# Create the prompt.
prompt = "Summarize the speech."

# Pass the prompt and the audio file to Gemini.
response = model.generate_content([prompt, audio_file])

# Print the response.
print(response.text)

Embed the audio file

Download a small audio file:

# Download an audio file from a remote server to a cache directory
# on your local host.
curl -o samplesmall.mp3 https://storage.googleapis.com/generativeai-downloads/data/Apollo-11_Day-01-Highlights-10s.mp3

Then, pass that downloaded small audio file along with the prompt to Gemini:

# Initialize a Gemini model appropriate for your use case.
model = genai.GenerativeModel('models/gemini-1.5-flash')

# Create the prompt.
prompt = "Please summarize the audio."

# Load the samplesmall.mp3 file into a Python Blob object containing the audio
# file's bytes and then pass the prompt and the audio to Gemini.
response = model.generate_content([
    prompt,
    {
        "mime_type": "audio/mp3",
        "data": pathlib.Path('samplesmall.mp3').read_bytes()
    }
])

# Output Gemini's response to the prompt and the inline audio.
print(response.text)

Inline Audio Data Guidelines

• Total request size limit: 20 MB (includes text prompts, instructions, and inline files)
• Exceeding 20 MB? Use the File API for uploading files instead
• Reusing an audio sample? Uploading via File API is more efficient

More Audio Capabilities – Gemini API

• Get a Transcript: Simply request a transcript in your prompt to receive a text version of the audio content.

# Initialize a Gemini model appropriate for your use case.
model = genai.GenerativeModel(model_name="gemini-1.5-flash")

# Create the prompt.
prompt = "Generate a transcript of the speech."

# Pass the prompt and the audio file to Gemini.
response = model.generate_content([prompt, audio_file])

# Print the transcript.
print(response.text)

Reference Audio Timestamps: Specify timestamps in your prompt to focus on specific sections of the audio file. Use the format MM: SS to indicate start and end times.

# Create a prompt containing timestamps.
prompt = "Provide a transcript of the speech from 02:30 to 03:29."

This prompt requests a transcript of the audio section starting at 2 minutes 30 seconds and ending at 3 minutes 29 seconds.

• Count tokens: Call the countTokens method to get a count of the number of tokens in the audio file.

model.count_tokens([audio_file])

Gemini Flash – Gemini API

Built for speed

• Sub-second average first-token latency for most developer and enterprise use cases.

Longer context

• Flash has a one-million-token context window by default, which means you can process one hour of video, 11 hours of audio, and codebases with more than 30,000 lines of code, or over 700,000 words.

Relentless innovation

• Our research team continually explores new ideas at the frontier of AI, building innovative products that show consistent progress on a range of benchmarks. Our latest model is Gemini 1.5 Flash.