Running Managed Open‑Source Models on VertexAI

When I first looked at VertexAI I thought it only hosted closed models.
I was wrong.
VertexAI also serves managed open source models.
You can call them directly from your code or through the Gemini CLI.

The workflow is simple.
Start by exporting a few environment variables.

export GOOGLE_GEMINI_BASE_URL="http://localhost:4000"
export GEMINI_API_KEY=sk-1234567890
export GCP_ACCESS_TOKEN=$(gcloud auth print-access-token)

Next create a virtual environment.
I like uv because it is fast.

uv venv
source .venv/bin/activate
uv pip install 'litellm[proxy]'

Now write a config.yaml.
Below is a minimal example that points at a model called zai-org/glm-5-maas.

model_list:
  - model_name: zai-org/glm-5-maas
    litellm_params:
      model: openai/zai-org/glm-5-maas
      api_base: https://aiplatform.googleapis.com/v1/projects/project-id/locations/global/endpoints/openapi/
      api_key: "os.environ/GCP_ACCESS_TOKEN"

router_settings:
  model_group_alias: {"gemini-2.5-pro": "zai-org/glm-5-maas", "gemini-2.5-flash": "zai-org/glm-5-maas"}

With this file in place you can start the Gemini CLI.

gemini

The CLI will read the config, build the request URL, and send it to the LiteLLM proxy running on localhost:4000.
The proxy then forwards the call to VertexAI using the access token you supplied.

A few things to notice:

• The proxy can balance traffic across several models.
• If one endpoint fails the request falls back to another.
• Cost tracking is built‑in, so you can see how many dollars you spend per million tokens.

I made a typo in my first config.yaml.
The CLI crashed with a permission error.
After fixing the typo the request worked on the first try.
Small mistakes happen.

If you want to use a non‑Gemini model you need to add a model_group_alias.
For example, to route gemini-2.5-pro to GPT‑4 you can add:

model_list:
  - model_name: gpt-4o-model
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  model_group_alias: {"gemini-2.5-pro": "gpt-4o-model"}

This tells the proxy that any request for gemini-2.5-pro should be sent to the GPT‑4 endpoint.
The same pattern works for Anthropic, Bedrock, or any provider supported by LiteLLM.

Troubleshooting tips:

• Verify that the proxy is reachable at the URL you set in GOOGLE_GEMINI_BASE_URL.
• Double check that the API key environment variable is exported correctly.
• Run npm install again if you see build errors after pulling the Gemini CLI repository.

In my view the biggest win is the single place to manage access.
You no longer need to hand out separate keys to developers.
You can set a monthly budget and watch the numbers climb in real time.
That level of control is rare in the open source world.

Give it a try.
Follow the steps above and you should see responses within seconds.
If something goes wrong, check the logs of the proxy; they usually point to a missing env var or a typo.

Happy hacking!