Re: Linux Loopback Interface

From: Tanel Poder <tanel_at_tanelpoder.com>
Date: Wed, 11 Sep 2019 22:53:28 -0400
Message-ID: <CAMHX9JKY4r-hMWKXSPoVknm0s-4w6ZhA1KPUHQbBhxYrqPRm6g_at_mail.gmail.com>

You could use lsof to see who's using any TCP/UDP sockets with IP 127.0.0.1 as one endpoint (or any other IP if any have been assigned to loopback interface)

$ *sudo lsof -nPi | grep 127.0.0.1*

chronyd    5334 chrony    1u  IPv4   35307      0t0  UDP 127.0.0.1:323
master     6535   root   13u  IPv4   38297      0t0  TCP 127.0.0.1:25

(LISTEN) Regarding the memory errors and reducing MTU size, there have been similar issues with jumbo frames. "Large" frames and MTU sizes require kernel to allocate large *contiguous* memory chunks from kernel memory. Since such allocation sizes can vary (different kernel drivers & allocation reasons using them), this kernel memory can get fragmented, thus finding a "large" chunk on demand gets problematic. This can cause paging/reclamation issues (even if there's some memory free) or kernel memory allocation failures.

One solution would be to tune the Linux kernel settings to keep more memory free at any time (vm.min_free_kbytes) or use lower MTU/frame/kernel memory allocation sizes.

You can monitor the fragmentation of (some) kernel memory via */proc/buddyinfo*. This is a good article explaining the buddy allocator basics and how to read /proc/buddyinfo + a python script:

$ *cat /proc/buddyinfo *

Node 0, zone      DMA      1      0      0      0      2      1      1
 0      1      1      3
Node 0, zone    DMA32   2656   4401   3224   4288   1480    277     29
 0      0      0      0
Node 0, zone   Normal     23     76     25     16      5      0      0
 0      1      0      0

$ *./buddyinfo.py *
Node: 0
 Zone: DMA
 Free KiB in zone: 15876.00

Fragment size        Free fragments       Total available KiB
4096                 1                    4.0
8192                 0                    0.0
16384                0                    0.0
32768                0                    0.0
65536                2                    128.0
131072               1                    128.0
262144               1                    256.0
524288               0                    0.0
1048576              1                    1024.0
2097152              1                    2048.0
4194304              3                    12288.0

 Zone: DMA32
 Free KiB in zone: 368760.00

Fragment size        Free fragments       Total available KiB
4096                 1788                 7152.0
8192                 4401                 35208.0
16384                3224                 51584.0
32768                4288                 137216.0
65536                1480                 94720.0
131072               277                  35456.0
262144               29                   7424.0
524288               0                    0.0
1048576              0                    0.0
2097152              0                    0.0
4194304              0                    0.0

 Zone: Normal
 Free KiB in zone: 2956.00

Fragment size        Free fragments       Total available KiB
4096                 23                   92.0
8192                 76                   608.0
16384                25                   400.0
32768                16                   512.0
65536                5                    320.0
131072               0                    0.0
262144               0                    0.0
524288               0                    0.0
1048576              1                    1024.0
2097152              0                    0.0
4194304              0                    0.0

Thanks

--
Tanel Poder
https://blog.tanelpoder.com/seminar


On Tue, Sep 10, 2019 at 10:20 PM Rakesh Ra <rakeshra.tr_at_gmail.com> wrote:



> Hi All,


>

> Can someone shed some light on for what operations Oracle can use loopback

> interface? We had the below issue on one of our primary production database

> running on 12.1.0.2 running on Red Hat Linux 7.5.

>

>

> skgxpvfynet: mtype: 61 process 75262 failed because of a resource problem

> in the OS. The OS has most likely run out of buffers (rval: 4) opiodr

> aborting process unknown ospid (75262) as a result of ORA-603

>

> Oracle asked us to fix the MTU by lowering it from 64k to 16k by following

> the below note.There were just application user connections (jsut less than

> 400 user processes) and couple of scp process which was shipping the backup

> files from the this primary database server to standby database server as

> part of DR build.

>

> lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 16436  <<<<<<<<<<<< This value was

> 64K during the issue time. It looks starting 6.5 kernel , MTU is bumped to

> 64K. We lowered this to 16K

>         inet 127.0.0.1  netmask 255.0.0.0

>         loop  txqueuelen 1000  (Local Loopback)

>         RX packets 2689657  bytes 17458642228 (16.2 GiB)

>         RX errors 0  dropped 0  overruns 0  frame 0

>         TX packets 2689657  bytes 17458642228 (16.2 GiB)

>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

>

> Oracle Linux: ORA-27301:OS Failure Message: No Buffer Space Available (Doc

> ID 2041723.1)

>

> I followed up with Oracle to see, in what way loopback interface is

> contributing to the issue here, but they failed to give more details saying

> it's OS specific and they don't have idea on the same. :(

>

> Appreciate if someone has some previous experience working on similar

> issue provide some information related to this.

>

> BR,

> Rakesh RA

>


--
http://www.freelists.org/webpage/oracle-l

Received on Thu Sep 12 2019 - 04:53:28 CEST