linux-stable.git/include/net/inet_hashtables.h, branch linux-4.6.y

soreuseport: fast reuseport TCP socket selection

2016-02-11T08:54:15+00:00

This change extends the fast SO_REUSEPORT socket lookup implemented
for UDP to TCP.  Listener sockets with SO_REUSEPORT and the same
receive address are additionally added to an array for faster
random access.  This means that only a single socket from the group
must be found in the listener list before any socket in the group can
be used to receive a packet.  Previously, every socket in the group
needed to be considered before handing off the incoming packet.

This feature also exposes the ability to use a BPF program when
selecting a socket from a reuseport group.

Signed-off-by: Craig Gallek 
Signed-off-by: David S. Miller

inet: refactor inet[6]_lookup functions to take skb

2016-02-11T08:54:14+00:00

This is a preliminary step to allow fast socket lookup of SO_REUSEPORT
groups.  Doing so with a BPF filter will require access to the
skb in question.  This change plumbs the skb (and offset to payload
data) through the call stack to the listening socket lookup
implementations where it will be used in a following patch.

Signed-off-by: Craig Gallek 
Signed-off-by: David S. Miller

sock: struct proto hash function may error

2016-02-11T08:54:14+00:00

In order to support fast reuseport lookups in TCP, the hash function
defined in struct proto must be capable of returning an error code.
This patch changes the function signature of all related hash functions
to return an integer and handles or propagates this return value at
all call sites.

Signed-off-by: Craig Gallek 
Signed-off-by: David S. Miller

tcp/dccp: fix hashdance race for passive sessions

2015-10-23T12:42:21+00:00

Multiple cpus can process duplicates of incoming ACK messages
matching a SYN_RECV request socket. This is a rare event under
normal operations, but definitely can happen.

Only one must win the race, otherwise corruption would occur.

To fix this without adding new atomic ops, we use logic in
inet_ehash_nolisten() to detect the request was present in the same
ehash bucket where we try to insert the new child.

If request socket was not found, we have to undo the child creation.

This actually removes a spin_lock()/spin_unlock() pair in
reqsk_queue_unlink() for the fast path.

Fixes: e994b2f0fb92 ("tcp: do not lock listener to process SYN packets")
Fixes: 079096f103fa ("tcp/dccp: install syn_recv requests into ehash table")
Signed-off-by: Eric Dumazet 
Signed-off-by: David S. Miller

tcp/dccp: install syn_recv requests into ehash table

2015-10-03T11:32:41+00:00

In this patch, we insert request sockets into TCP/DCCP
regular ehash table (where ESTABLISHED and TIMEWAIT sockets
are) instead of using the per listener hash table.

ACK packets find SYN_RECV pseudo sockets without having
to find and lock the listener.

In nominal conditions, this halves pressure on listener lock.

Note that this will allow for SO_REUSEPORT refinements,
so that we can select a listener using cpu/numa affinities instead
of the prior 'consistent hash', since only SYN packets will
apply this selection logic.

We will shrink listen_sock in the following patch to ease
code review.

Signed-off-by: Eric Dumazet 
Cc: Ying Cai 
Cc: Willem de Bruijn 
Signed-off-by: David S. Miller

inet: constify __inet_inherit_port() sock argument

2015-09-29T23:53:08+00:00

socket is not touched, make it const.

Signed-off-by: Eric Dumazet 
Signed-off-by: David S. Miller

inet: simplify timewait refcounting

2015-07-09T22:12:20+00:00

timewait sockets have a complex refcounting logic.
Once we realize it should be similar to established and
syn_recv sockets, we can use sk_nulls_del_node_init_rcu()
and remove inet_twsk_unhash()

In particular, deferred inet_twsk_put() added in commit
13475a30b66cd ("tcp: connect() race with timewait reuse")
looks unecessary : When removing a timewait socket from
ehash or bhash, caller must own a reference on the socket
anyway.

Signed-off-by: Eric Dumazet 
Signed-off-by: David S. Miller

tcp: fix/cleanup inet_ehash_locks_alloc()

2015-05-26T23:48:46+00:00

If tcp ehash table is constrained to a very small number of buckets
(eg boot parameter thash_entries=128), then we can crash if spinlock
array has more entries.

While we are at it, un-inline inet_ehash_locks_alloc() and make
following changes :

- Budget 2 cache lines per cpu worth of 'spinlocks'
- Try to kmalloc() the array to avoid extra TLB pressure.
  (Most servers at Google allocate 8192 bytes for this hash table)
- Get rid of various #ifdef

Signed-off-by: Eric Dumazet 
Signed-off-by: David S. Miller

inet_hashinfo: remove bsocket counter

2015-05-21T22:55:32+00:00

We no longer need bsocket atomic counter, as inet_csk_get_port()
calls bind_conflict() regardless of its value, after commit
2b05ad33e1e624e ("tcp: bind() fix autoselection to share ports")

This patch removes overhead of maintaining this counter and
double inet_csk_get_port() calls under pressure.

Signed-off-by: Eric Dumazet 
Cc: Marcelo Ricardo Leitner 
Cc: Flavio Leitner 
Acked-by: Flavio Leitner 
Signed-off-by: David S. Miller

inet: get rid of last __inet_hash_connect() argument

2015-03-19T02:00:35+00:00

We now always call __inet_hash_nolisten(), no need to pass it
as an argument.

Signed-off-by: Eric Dumazet 
Signed-off-by: David S. Miller