Hello,
I was searching for other ways to run using small LLMs for specific tasks like private web searching. I found a low-memory-footprint LLM solution using LiteRT-LM(formerly tensorflow lite) from Google: https://ai.google.dev/edge/litert-lm/cli
It initializes the model faster than vllm. It can also be used in combination with other LLMs on a single Jetson or DGX Spark system, allowing multiple LLMs to run simultaneously on the same device.
I used it together with a SearXNG meta-search setup. More details can be found here: Playbook #1 – Open WebUI + SearXNG (Private Web Search) on DGX Spark
Create a virtual environment:
uv venv .litert --python 3.12
source .litert/bin/activate
Then install LiteRT-LM:
uv pip install --upgrade litert-lm
Run the model using:
litert-lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
gemma-4-E2B-it.litertlm \
--prompt="What can you tell me about quantum computing?"
With MTP enabled:
litert-lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
gemma-4-E2B-it.litertlm \
--backend=gpu \
--enable-speculative-decoding=true \
--preset=preset.py
Output:
> When was Gemini 3.5 Flash launched?
[tool_call] {"name": "web_search", "arguments": {"query": "Gemini 3.5 Flash launch date"}}
[tool_response] "Search results for: Gemini 3.5 Flash launch date\n\n1. Google launches Gemini 3.5 Flash. How to try it for free. | Mashable\n URL: https://mashable.com/article/google-io-2026-gemini-35-flash\n 24 \u0441\u0430\u0493\u0430\u0442 \u0431\u04b1\u0440\u044b\u043d \u00b7 Google unveiled a new AI model, Gemini 3.5 Flash, at Google I/O 2026. The new model is available now at no cost.\n\n2. Gemini 3.5: frontier intelligence with action - Google Blog\n URL: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/\n Today, we\u2019re introducing Gemini 3.5, our latest family of models combining frontier intelligence with action. This represents a major leap forward in building more capable, intelligent agents. We\u2019re kicking off the series by releasing 3.5 Flash.\n\n3. Gemini 3.5 Flash: Benchmarks, Pricing, and Complete Specs\n URL: https://llm-stats.com/blog/research/gemini-3.5-flash-launch\n Google released Gemini 3.5 Flash today at Google I/O 2026. It is the first model in the new Gemini 3.5 family and the strongest agentic and coding model the Flash series has ever shipped.\n\n4. Gemini 3.5 Pro Is Coming Next Month \u2014 What the Flash Release Already Tells Us | WaveSpeed Blog\n URL: https://wavespeed.ai/blog/posts/gemini-3-5-pro-coming-next-month/\n It\u2019s 3.5. But the more interesting part is what Google actually launched and what it held back: Gemini 3.5 Flash shipped to general availability on May 19; Gemini 3.5 Pro is \u201ccoming next month.\u201d Sundar Pichai\u2019s exact line on stage: \u201cGive ...\n\n5. Google Introduces Gemini 3.5 Flash at I/O 2026: A Faster and Cheaper ...\n URL: https://www.marktechpost.com/2026/05/20/google-introduces-gemini-3-5-flash-at-i-o-2026-a-faster-and-cheaper-model-for-ai-agents-and-coding/\n Google releases Gemini 3.5 Flash at I/O 2026, outperforming Gemini 3.1 Pro on coding and agentic benchmarks at 4x the speed."
The search results indicate that **Gemini 3.5 Flash was unveiled at Google I/O 2026**.
One result mentions that it was available now at no cost, and another suggests it shipped to general availability on **May 19**. Please note that the search results point to an event in **2026**, which suggests this information might be from a future event or a misunderstanding of the current timeline.
For the most accurate and up-to-date information, I recommend checking the official Google AI or Google DeepMind announcements.
Another example output:
> What was score between arsenal vs atlético madrid in 2026?
[tool_call] {"name": "web_search", "arguments": {"query": "Arsenal vs Atletico Madrid 2026 score"}}
[tool_response] "Search results for: Arsenal vs Atletico Madrid 2026 score\n\n1. Arsenal vs Atl\u00e9tico Madrid live score, H2H and lineups - Sofascore\n URL: https://www.sofascore.com/football/match/atletico-madrid-arsenal/RsLgb\n Arsenal Atl\u00e9tico Madrid live score starts on 5 May 2026 at 19:00 UTC time at Emirates Stadium stadium, London city, England in UEFA Champions League, Knockout stage, Europe.\n\n2. Arsenal 1-0 Atl\u00e9tico (May 5, 2026) Final Score - ESPN\n URL: https://www.espn.com/soccer/match/_/gameId/401862896/atletico-madrid-arsenal\n Bukayo Saka scored the decisive goal to send Arsenal to their first Champions League final in 20 years following a 1-0 win against Atletico Madrid at the ...\n\n3. Atl\u00e9tico Madrid vs. Arsenal - Final Score - April 29, 2026\n URL: https://www.foxsports.com/soccer/champions-league-atletico-madrid-vs-arsenal-apr-29-2026-game-boxscore-710731\n View the Atl\u00e9tico Madrid vs. Arsenal game played on April 29, 2026. Box score, stats, play-by-play timeline, highlights, odds & more\n\n4. Arsenal vs Atletico Madrid - live score, predicted lineups and H2H stats\n URL: https://www.fotmob.com/matches/arsenal-vs-atletico-madrid/37wkxc\n Arsenal vs Atletico Madrid on Tue, May 5, 2026, 19:00 UTC. Check live results, H2H, match stats, lineups, player ratings, insights, team forms, shotmap, and highlights.\n\n5. Atl\u00e9tico 1-1 Arsenal (Apr 29, 2026) Final Score - ESPN\n URL: https://www.espn.com/soccer/match/_/gameId/401862894/arsenal-atletico-madrid\n Match ends, Atletico Madrid 1, Arsenal 1. 90'+7'. Second Half ends, Atletico Madrid 1, Arsenal 1. 90'+7'. Foul by Robin Le Normand (Atletico Madrid). Full ..."
Based on the search results, there were two matches mentioned for Arsenal vs. Atlético Madrid in 2026:
1. **April 29, 2026:** The final score was **Atlético Madrid 1 - 1 Arsenal**.
2. **May 5, 2026:** The final score was **Arsenal 1 - 0 Atlético Madrid**.
Please specify which match you are interested in if you need more details!
>