Running machine learning models completely offline on Android gives you zero latency, zero API costs, and full user privacy. Google's LiteRT (com.google.ai.edge.litert, the evolution of TensorFlow Lite) makes it straightforward to run .tflite models directly on-device.

In this tutorial, we'll walk through how NutriLens — a Jetpack Compose + CameraX food & nutrition tracker — uses LiteRT to identify dishes from a live camera feed and estimate calories, macros, and micronutrients 100% offline.

What We Are Building

The NutriLens scanning pipeline works like this:

CameraX ImageAnalysis (RGBA_8888) │ ▼ Rotate upright & throttle (~850ms) │ ▼ Center-crop & scale Bitmap (192×192) │ ▼ LiteRT Inference (CompiledModel / Interpreter) │ ▼ Dequantize UINT8 Softmax → Top-3 Food Labels │ ▼ Temporal Smoothing + Nutrition Estimator → Compose UI 

Step 1: Add LiteRT and CameraX Dependencies

In gradle/libs.versions.toml:

[versions] camerax = "1.4.2" litert = "2.1.0" [libraries] androidx-camera-core = { group = "androidx.camera", name = "camera-core", version.ref = "camerax" } androidx-camera-camera2 = { group = "androidx.camera", name = "camera-camera2", version.ref = "camerax" } androidx-camera-lifecycle = { group = "androidx.camera", name = "camera-lifecycle", version.ref = "camerax" } androidx-camera-view = { group = "androidx.camera", name = "camera-view", version.ref = "camerax" } google-ai-edge-litert = { group = "com.google.ai.edge.litert", name = "litert", version.ref = "litert" } 

In app/build.gradle.kts, add the dependencies and disable compression for .tflite assets so Android can memory-map (mmap) the model file directly without decompressing it into RAM:

android { // ... androidResources { noCompress += "tflite" } } dependencies { implementation(libs.androidx.camera.core) implementation(libs.androidx.camera.camera2) implementation(libs.androidx.camera.lifecycle) implementation(libs.androidx.camera.view) implementation(libs.google.ai.edge.litert) } 

Tip: Never skip noCompress += "tflite". If Gradle compresses your .tflite asset inside the APK, assets.openFd(...) will fail and prevent zero-copy memory mapping.

Step 2: Bundle Your Offline .tflite Model and Labels

Place your model and label files inside app/src/main/assets/:

app/src/main/assets/ ├── vision_classifier_food.tflite # aiy/vision/classifier/food_V1 (2,024 classes) └── food_labels.txt # 2,024 class labels matching output indices 

Before writing Kotlin code, inspect your model's input and output tensor specs:

  • Input: [1, 192, 192, 3] (UINT8 RGB pixels 0..255)
  • Output: [1, 2024] (UINT8 quantized softmax probabilities)

Knowing whether your model uses UINT8 or FLOAT32 is critical — feeding FLOAT32 normalized data to a UINT8 model (or forgetting to dequantize UINT8 outputs) is one of the most common reasons an offline model outputs garbage.

Step 3: Initialize LiteRT (CompiledModel + Interpreter Fallback)

LiteRT 2.1.0 provides the modern CompiledModel + TensorBuffer API (com.google.ai.edge.litert.*) as well as a compatible Interpreter (org.tensorflow.lite.Interpreter bundled inside the litert artifact).

Older .tflite models (like aiy/vision/classifier/food_V1) sometimes lack SignatureDefs or use UINT8 quantization parameters (scale and zeroPoint). Initializing CompiledModel first and keeping Interpreter ready for tensor metadata & fallback gives you the best of both worlds:

class LiteRtFoodClassifier( private val context: Context ) { private var compiledModel: CompiledModel? = null private var inputBuffers: List<TensorBuffer>? = null private var outputBuffers: List<TensorBuffer>? = null private var interpreter: Interpreter? = null private var inputWidth = 192 private var inputHeight = 192 private var inputDataType = DataType.UINT8 private var outputClassCount = 2024 private var outputDataType = DataType.UINT8 private var outputScale = 1f / 255f private var outputZeroPoint = 0 private var labels: List<String> = emptyList() @Synchronized fun ensureInitialized(): Boolean { if ((compiledModel != null || interpreter != null) && labels.isNotEmpty()) { return true } labels = loadLabelsFromAssets() // 1. Modern LiteRT CompiledModel API val compiledReady = runCatching { val model = CompiledModel.create( context.assets, "vision_classifier_food.tflite", CompiledModel.Options(Accelerator.CPU) ) val inputs = model.createInputBuffers() val outputs = model.createOutputBuffers() if (inputs.isNotEmpty() && outputs.isNotEmpty()) { compiledModel = model inputBuffers = inputs outputBuffers = outputs true } else { model.close() false } }.getOrDefault(false) // 2. Inspect tensor shapes, DataTypes, and quantization params via Interpreter val interpreterReady = runCatching { val modelBuffer = loadMappedModelBuffer("vision_classifier_food.tflite") val tflite = Interpreter(modelBuffer, Interpreter.Options().apply { setNumThreads(4) }) val inTensor = tflite.getInputTensor(0) val inShape = inTensor.shape() // [1, 192, 192, 3] if (inShape.size >= 3) { inputHeight = inShape[1] inputWidth = inShape[2] } inputDataType = inTensor.dataType() val outTensor = tflite.getOutputTensor(0) outputClassCount = outTensor.shape().lastOrNull() ?: 2024 outputDataType = outTensor.dataType() val qParams = outTensor.quantizationParams() if (qParams.scale > 0f) { outputScale = qParams.scale outputZeroPoint = qParams.zeroPoint } interpreter = tflite true }.getOrDefault(false) return compiledReady || interpreterReady } } 

Memory-mapping the model from assets:

private fun loadMappedModelBuffer(assetName: String): ByteBuffer { return runCatching { context.assets.openFd(assetName).use { fd -> FileInputStream(fd.fileDescriptor).channel.use { channel -> channel.map( FileChannel.MapMode.READ_ONLY, fd.startOffset, fd.declaredLength ) } } }.getOrElse { val bytes = context.assets.open(assetName).use { it.readBytes() } ByteBuffer.allocateDirect(bytes.size).apply { order(ByteOrder.nativeOrder()) put(bytes) rewind() } } } 

Step 4: Preprocess Bitmaps & Run Inference

Before passing a camera Bitmap into LiteRT:

  1. Convert Bitmap.Config.HARDWARE to ARGB_8888 if needed (since getPixels cannot read hardware bitmaps).
  2. Center-crop to a square and scale to 192×192.
  3. Extract packed RGB bytes (R, G, B) in row-major order.
private fun centerCropAndScale(source: Bitmap, targetWidth: Int, targetHeight: Int): Bitmap { val safeBitmap = if (source.config == Bitmap.Config.HARDWARE) { source.copy(Bitmap.Config.ARGB_8888, false) } else { source } val minDim = min(safeBitmap.width, safeBitmap.height) val cropX = (safeBitmap.width - minDim) / 2 val cropY = (safeBitmap.height - minDim) / 2 val cropped = Bitmap.createBitmap(safeBitmap, cropX, cropY, minDim, minDim) return if (cropped.width == targetWidth && cropped.height == targetHeight) { cropped } else { Bitmap.createScaledBitmap(cropped, targetWidth, targetHeight, true) } } 

When reading quantized UINT8 output bytes in Kotlin/Java, remember that Byte is signed (-128..127), so always mask with and 0xFF and apply the quantization formula:

val probability = ((rawByte.toInt() and 0xFF) - outputZeroPoint) * outputScale 

Running inference with CompiledModel + TensorBuffer:

val model = compiledModel val inputs = inputBuffers val outputs = outputBuffers if (model != null && inputs != null && outputs != null) { val probs = runCatching { val rgbBytes = bitmapToRgbByteArray(croppedBitmap, inputWidth, inputHeight) if (inputDataType == DataType.FLOAT32) { val floatIn = FloatArray(rgbBytes.size) { i -> ((rgbBytes[i].toInt() and 0xFF) / 127.5f) - 1.0f } inputs[0].writeFloat(floatIn) model.run(inputs, outputs) outputs[0].readFloat() } else { inputs[0].writeInt8(rgbBytes) model.run(inputs, outputs) if (outputDataType == DataType.FLOAT32) { outputs[0].readFloat() } else { val rawOut = outputs[0].readInt8() FloatArray(rawOut.size) { i -> (((rawOut[i].toInt() and 0xFF) - outputZeroPoint) * outputScale).coerceAtLeast(0f) } } } }.getOrNull() } 

Always release TensorBuffer, CompiledModel, and Interpreter resources when your ViewModel clears (onCleared()).

Step 5: Stream Live Frames from CameraX

In CameraPreviewView.kt, configure ImageAnalysis with OUTPUT_IMAGE_FORMAT_RGBA_8888 and STRATEGY_KEEP_ONLY_LATEST so slow frames never queue up. Don't forget to rotate the Bitmap using imageProxy.imageInfo.rotationDegrees — otherwise your model receives sideways images and accuracy drops sharply:

val imageAnalysis = ImageAnalysis.Builder() .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) .build() imageAnalysis.setAnalyzer(analysisExecutor) { imageProxy -> try { val rawBitmap = imageProxy.toBitmap() val rotation = imageProxy.imageInfo.rotationDegrees val uprightBitmap = if (rotation != 0) { val matrix = Matrix().apply { postRotate(rotation.toFloat()) } Bitmap.createBitmap( rawBitmap, 0, 0, rawBitmap.width, rawBitmap.height, matrix, true ) } else { rawBitmap } onFrameBitmap(uprightBitmap) } finally { imageProxy.close() } } 

Step 6: Stabilize Live Detections in the ViewModel

Running inference on every camera frame can cause flickering or stale UI states if you aren't careful. Here are three practical rules we applied in ScanFoodViewModel:

  1. Throttle & guard concurrency: Skip incoming frames if less than 850ms has passed or if an AtomicBoolean (isClassifyingFrame) is already active. Run inference on Dispatchers.Default.
  2. Use honest confidence thresholds: A 2,024-class softmax spreads probability across many labels, so real top-1 confidence on a good match is often 28%–65%, not 90%+. Never artificially inflate confidence scores in code; instead calibrate your lock threshold (MIN_LIVE_CONFIDENCE_PERCENT = 28) to the model's real distribution.
  3. Smooth transitions instead of hard-locking:
    • First match (>= 28%) updates the UI right away.
    • Switching to a different food requires 2 consecutive frames (pendingCandidateStreak >= 2) so brief camera shake doesn't overwrite a good detection.
    • Pointing away from food (< 18% for 3 consecutive frames) clears the previous item so the user can seamlessly scan a new plate.

Step 7: Map Detected Labels to Offline Nutrition Data

Once LiteRT returns a food label (e.g., "Mango Sticky Rice", "Breakfast Burrito"), FoodNutritionEstimator maps the label offline to calories, protein/carbs/fat, serving weight, and micronutrients (fiber, sugar, sodium, potassium, calcium, iron, vitamin C, cholesterol). Users can also tap any macro card, adjust serving multipliers (0.5x, 1.0x, 1.5x, or custom grams), or switch between the top-3 predicted matches before logging to Room/SQLite.

Key Takeaways & Common Pitfalls

  • Always inspect your .tflite tensors: Check input size (192×192), channel order (RGB), and quantization (UINT8 vs FLOAT32) before writing preprocessing code.
  • Avoid overriding fine-grained classifiers with coarse segmenters: If you combine a classifier with a semantic segmenter, let the whole-image classifier name cohesive dishes instead of forcing segment categories into hardcoded fallback names.
  • Reuse direct ByteBuffers for heavy outputs: Large segmentation tensors (like [1, 513, 513, 26] FLOAT32 ≈ 27 MB) should reuse a cached direct ByteBuffer rather than allocating per frame.
  • Keep live scanning responsive: Use STRATEGY_KEEP_ONLY_LATEST, throttle frames, and use short multi-frame streaks instead of permanently pausing the camera after the first detection.

With LiteRT + CameraX, you get a fast, privacy-first, zero-cost AI scanner that works anywhere — even in airplane mode.
Github: https://github.com/tusharahmed11/NutriLens