Function drm_malloc_ab() returns a NULL pointer if there is no enough memory. In function vc4_cl_lookup_bos() defined in file file drivers/gpu/drm/vc4/vc4_gem.c, varaible ret takes the return error code. When the call to drm_malloc_ab() (at line 545) fails, however, ret is not set to a negative integer, and as a result, 0 will be returned. 0 indicates that there is no error, which is contrary to the fact. Maybe it is better to assign "-ENOMEM" to ret when the check of the return value of drm_malloc_ab() fails. Codes related to this bug are summarised as follows.

vc4_cl_lookup_bos @@ drivers/gpu/drm/vc4/vc4_gem.c
518 static int
519 vc4_cl_lookup_bos(struct drm_device *dev,
520           struct drm_file *file_priv,
521           struct vc4_exec_info *exec)
522 {
523     struct drm_vc4_submit_cl *args = exec->args;
524     uint32_t *handles;
525     int ret = 0;
        ...
538     exec->bo = drm_calloc_large(exec->bo_count,
539                     sizeof(struct drm_gem_cma_object *));
540     if (!exec->bo) {
541         DRM_ERROR("Failed to allocate validated BO pointers\n");
542         return -ENOMEM;
543     }
544 
545     handles = drm_malloc_ab(exec->bo_count, sizeof(uint32_t));
546     if (!handles) {
547         DRM_ERROR("Failed to allocate incoming GEM handles\n");
548         goto fail;       // insert "ret = -ENOMEM;" before this jump instruction?
549     }
        ...
575 fail:
576     drm_free_large(handles);
577     return ret;
578 }

Thanks very much!