Hello.

There is a wrong system call used for compatibility between native kernel and O32 userspace:
/arch/mips/kernel/scall64-o32.S

PTR sys_getsockopt
but:
PTR compat_sys_setsockopt

This false symmetry in system calls leads to erroneous behavior when any data structure (which differs in size for 32-bit and 64-bit subsystems) is passed from O32 into kernel via setsockopt(), got back via getsockopt() and finally sent again via setsockopt().
First compat_sys_setsockopt will bloat it correctly for kernel space usage, then sys_getsockopt will send it to userspace without backward conversion, and even if userspace does nothing with this data, second compat_sys_setsockopt will bloat it again.

One of the easier ways to reproduce the consequences of the bug is to use iptables compiled for O32 with ULOG target. We will get the error on the second try:

First call
>iptables -I INPUT -j ULOG
(OK)
Second call
>iptables -I INPUT -j ULOG
x_tables: ip_tables: ULOG.0 target: invalid size 56 (kernel) != (user) 64

The second call does not matter a bit here, as it is the first rule that gets bloated twice in kernel, so we'll get the error as long as the first ULOG rule succeeds.

Also, there was a patch fixing this bug, but only concerning N32:
https://patchwork.linux-mips.org/patch/7115/

To fix the bug, it is needed to reapply the patch above for O32 (scall64-o32.S)
-	PTR	sys_getsockopt
+	PTR	compat_sys_getsockopt