Bug 33992

Summary: Applying inappropriate ioctl operation on socket should return ENOTTY
Product: Networking Reporter: Lifeng Sun (lifongsun)
Component: OtherAssignee: Arnaldo Carvalho de Melo (acme)
Status: RESOLVED CODE_FIX    
Severity: normal CC: florian, jprobeau
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: all Subsystem:
Regression: No Bisected commit-id:
Attachments: patch

Description Lifeng Sun 2011-04-27 04:02:54 UTC
ioctl() calls against a socket with an inappropriate ioctl operation
are incorrectly returning EINVAL rather than ENOTTY:

  [ENOTTY]
      Inappropriate I/O control operation.

This bug causes that when the standard input/output/error of ncurses
are redirected to a socket, it fails to set a NOTTY flag hence sticks
on line-buffered mode,

/* ncurses-5.9/ncurses/tinfo/lib_ttyflags.c */
[...]
            for (;;) {
                if (SET_TTY(termp->Filedes, buf) != 0) {
                    if (errno == EINTR)
                        continue;
                    if ((errno == ENOTTY) && (SP_PARM != 0)) /* fail */
                        SP_PARM->_notty = TRUE;
                    result = ERR;
                }
                break;
            }
[...]

This bug is not limited to socket, it also occurs in a lot of, maybe some
hundred, other ioctl operations, while in the patch I only fixed about a
dozen of additional ones in pipe, fifo and character device drivers.
Comment 1 Lifeng Sun 2011-04-27 04:04:49 UTC
Created attachment 55612 [details]
patch
Comment 2 Florian Mickler 2011-05-04 06:09:34 UTC
A patch referencing this bug report has been merged in v2.6.39-rc5-274-g609cfda:

commit 41c31f318a5209922d051e293c61e4724daad11c
Author: Lifeng Sun <lifongsun@gmail.com>
Date:   Wed Apr 27 22:04:51 2011 +0000

    networking: inappropriate ioctl operation should return ENOTTY
Comment 3 jprobeau 2014-10-16 22:00:23 UTC
How would I install/ apply this? This patch fixes a problem, nut I am not sure how to apply/ install it?