Lambda Python runtime environment

python3.7

Response:
0

Request ID:
"50b03c17-4f23-4dee-b449-2d359f7b4e14"

Function Logs:
START RequestId: 50b03c17-4f23-4dee-b449-2d359f7b4e14 Version: $LATEST
===[environment-variables]===
PATH
LD_LIBRARY_PATH
LANG
TZ
LAMBDA_TASK_ROOT
LAMBDA_RUNTIME_DIR
AWS_REGION
AWS_DEFAULT_REGION
AWS_LAMBDA_LOG_GROUP_NAME
AWS_LAMBDA_LOG_STREAM_NAME
AWS_LAMBDA_FUNCTION_NAME
AWS_LAMBDA_FUNCTION_MEMORY_SIZE
AWS_LAMBDA_FUNCTION_VERSION
_AWS_XRAY_DAEMON_ADDRESS
_AWS_XRAY_DAEMON_PORT
AWS_XRAY_DAEMON_ADDRESS
AWS_XRAY_CONTEXT_MISSING
AWS_EXECUTION_ENV
_HANDLER
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
_X_AMZN_TRACE_ID

===[runtime directory:/var/runtime]===
python_dateutil-2.7.5.dist-info
docutils-0.14.dist-info
lambda_runtime_marshaller.py
s3transfer-0.1.13.dist-info
test_bootstrap.py
runtime-release
bootstrap.py
six-1.11.0.dist-info
urllib3-1.24.1.dist-info
s3transfer
lambda_runtime_exception.py
botocore
dateutil
bin
docutils
six.py
botocore-1.12.42.dist-info
urllib3
lambda_runtime_client.py
boto3-1.9.42.dist-info
boto3
bootstrap
test_lambda_runtime_client.py
jmespath
__pycache__
test_lambda_runtime_marshaller.py
jmespath-0.9.3.dist-info
===[amazon linux version]===
Amazon Linux AMI release 2017.03
0
END RequestId: 50b03c17-4f23-4dee-b449-2d359f7b4e14
REPORT RequestId: 50b03c17-4f23-4dee-b449-2d359f7b4e14  Duration: 109.71 ms Billed Duration: 200 ms     Memory Size: 128 MB Max Memory Used: 55 MB

Response:

Request ID:

"50b03c17-4f23-4dee-b449-2d359f7b4e14"

Function Logs:

START RequestId: 50b03c17-4f23-4dee-b449-2d359f7b4e14 Version: $LATEST

===[environment-variables]===

PATH

LD_LIBRARY_PATH

LANG

LAMBDA_TASK_ROOT

LAMBDA_RUNTIME_DIR

AWS_REGION

AWS_DEFAULT_REGION

AWS_LAMBDA_LOG_GROUP_NAME

AWS_LAMBDA_LOG_STREAM_NAME

AWS_LAMBDA_FUNCTION_NAME

AWS_LAMBDA_FUNCTION_MEMORY_SIZE

AWS_LAMBDA_FUNCTION_VERSION

_AWS_XRAY_DAEMON_ADDRESS

_AWS_XRAY_DAEMON_PORT

AWS_XRAY_DAEMON_ADDRESS

AWS_XRAY_CONTEXT_MISSING

AWS_EXECUTION_ENV

_HANDLER

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_SESSION_TOKEN

_X_AMZN_TRACE_ID

===[runtime directory:/var/runtime]===

python_dateutil-2.7.5.dist-info

docutils-0.14.dist-info

lambda_runtime_marshaller.py

s3transfer-0.1.13.dist-info

test_bootstrap.py

runtime-release

bootstrap.py

six-1.11.0.dist-info

urllib3-1.24.1.dist-info

s3transfer

lambda_runtime_exception.py

botocore

dateutil

bin

docutils

six.py

botocore-1.12.42.dist-info

urllib3

lambda_runtime_client.py

boto3-1.9.42.dist-info

boto3

bootstrap

test_lambda_runtime_client.py

jmespath

__pycache__

test_lambda_runtime_marshaller.py

jmespath-0.9.3.dist-info

===[amazon linux version]===

Amazon Linux AMI release 2017.03

END RequestId: 50b03c17-4f23-4dee-b449-2d359f7b4e14

REPORT RequestId: 50b03c17-4f23-4dee-b449-2d359f7b4e14 Duration: 109.71 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 55 MB

python3.6

Response:
{
  "errorMessage": "[Errno 2] No such file or directory: 'cat /etc/system-release': 'cat /etc/system-release'",
  "errorType": "FileNotFoundError",
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      22,
      "lambda_handler",
      "ret = subprocess.call(cmd)"
    ],
    [
      "/var/lang/lib/python3.6/subprocess.py",
      287,
      "call",
      "with Popen(*popenargs, **kwargs) as p:"
    ],
    [
      "/var/lang/lib/python3.6/subprocess.py",
      729,
      "__init__",
      "restore_signals, start_new_session)"
    ],
    [
      "/var/lang/lib/python3.6/subprocess.py",
      1364,
      "_execute_child",
      "raise child_exception_type(errno_num, err_msg, err_filename)"
    ]
  ]
}

Request ID:
"310363ce-1c33-4c07-9d31-9b445790422e"

Function Logs:
START RequestId: 310363ce-1c33-4c07-9d31-9b445790422e Version: $LATEST
===[environment-variables]===
PATH
LD_LIBRARY_PATH
LANG
TZ
LAMBDA_TASK_ROOT
LAMBDA_RUNTIME_DIR
AWS_REGION
AWS_DEFAULT_REGION
AWS_LAMBDA_LOG_GROUP_NAME
AWS_LAMBDA_LOG_STREAM_NAME
AWS_LAMBDA_FUNCTION_NAME
AWS_LAMBDA_FUNCTION_MEMORY_SIZE
AWS_LAMBDA_FUNCTION_VERSION
_AWS_XRAY_DAEMON_ADDRESS
_AWS_XRAY_DAEMON_PORT
AWS_XRAY_DAEMON_ADDRESS
AWS_XRAY_CONTEXT_MISSING
_X_AMZN_TRACE_ID
AWS_EXECUTION_ENV
_HANDLER
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_SECURITY_TOKEN
PYTHONPATH

===[runtime directory:/var/runtime]===
liblambdaruntime.so
docutils-0.14.dist-info
botocore-1.10.74.dist-info
s3transfer-0.1.13.dist-info
six-1.11.0.dist-info
boto3-1.7.74.dist-info
s3transfer
botocore
liblambdalog.so
dateutil
awslambda
docutils
six.py
boto3
liblambdaio.so
jmespath
__pycache__
python_dateutil-2.7.3.dist-info
jmespath-0.9.3.dist-info
liblambdaipc.so

===[amazon linux version]===
[Errno 2] No such file or directory: 'cat /etc/system-release': 'cat /etc/system-release': FileNotFoundError
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 22, in lambda_handler
    ret = subprocess.call(cmd)
  File "/var/lang/lib/python3.6/subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/var/lang/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/var/lang/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cat /etc/system-release': 'cat /etc/system-release'

END RequestId: 310363ce-1c33-4c07-9d31-9b445790422e
REPORT RequestId: 310363ce-1c33-4c07-9d31-9b445790422e  Duration: 50.25 ms  Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 49 MB

100

101

Response:

{

"errorMessage": "[Errno 2] No such file or directory: 'cat /etc/system-release': 'cat /etc/system-release'",

"errorType": "FileNotFoundError",

"stackTrace": [

[

"/var/task/lambda_function.py",

22,

"lambda_handler",

"ret = subprocess.call(cmd)"

[

"/var/lang/lib/python3.6/subprocess.py",

287,

"call",

"with Popen(*popenargs, **kwargs) as p:"

[

"/var/lang/lib/python3.6/subprocess.py",

729,

"__init__",

"restore_signals, start_new_session)"

[

"/var/lang/lib/python3.6/subprocess.py",

1364,

"_execute_child",

"raise child_exception_type(errno_num, err_msg, err_filename)"

]

}

Request ID:

"310363ce-1c33-4c07-9d31-9b445790422e"

Function Logs:

START RequestId: 310363ce-1c33-4c07-9d31-9b445790422e Version: $LATEST

===[environment-variables]===

PATH

LD_LIBRARY_PATH

LANG

LAMBDA_TASK_ROOT

LAMBDA_RUNTIME_DIR

AWS_REGION

AWS_DEFAULT_REGION

AWS_LAMBDA_LOG_GROUP_NAME

AWS_LAMBDA_LOG_STREAM_NAME

AWS_LAMBDA_FUNCTION_NAME

AWS_LAMBDA_FUNCTION_MEMORY_SIZE

AWS_LAMBDA_FUNCTION_VERSION

_AWS_XRAY_DAEMON_ADDRESS

_AWS_XRAY_DAEMON_PORT

AWS_XRAY_DAEMON_ADDRESS

AWS_XRAY_CONTEXT_MISSING

_X_AMZN_TRACE_ID

AWS_EXECUTION_ENV

_HANDLER

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_SESSION_TOKEN

AWS_SECURITY_TOKEN

PYTHONPATH

===[runtime directory:/var/runtime]===

liblambdaruntime.so

docutils-0.14.dist-info

botocore-1.10.74.dist-info

s3transfer-0.1.13.dist-info

six-1.11.0.dist-info

boto3-1.7.74.dist-info

s3transfer

botocore

liblambdalog.so

dateutil

awslambda

docutils

six.py

boto3

liblambdaio.so

jmespath

__pycache__

python_dateutil-2.7.3.dist-info

jmespath-0.9.3.dist-info

liblambdaipc.so

===[amazon linux version]===

[Errno 2] No such file or directory: 'cat /etc/system-release': 'cat /etc/system-release': FileNotFoundError

Traceback (most recent call last):

File "/var/task/lambda_function.py", line 22, in lambda_handler

ret = subprocess.call(cmd)

File "/var/lang/lib/python3.6/subprocess.py", line 287, in call

with Popen(*popenargs, **kwargs) as p:

File "/var/lang/lib/python3.6/subprocess.py", line 729, in __init__

restore_signals, start_new_session)

File "/var/lang/lib/python3.6/subprocess.py", line 1364, in _execute_child

raise child_exception_type(errno_num, err_msg, err_filename)

FileNotFoundError: [Errno 2] No such file or directory: 'cat /etc/system-release': 'cat /etc/system-release'

END RequestId: 310363ce-1c33-4c07-9d31-9b445790422e

REPORT RequestId: 310363ce-1c33-4c07-9d31-9b445790422e Duration: 50.25 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 49 MB

Python2.7

Response:
{
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      24,
      "lambda_handler",
      "ret = subprocess.call(cmd)"
    ],
    [
      "/usr/lib64/python2.7/subprocess.py",
      523,
      "call",
      "return Popen(*popenargs, **kwargs).wait()"
    ],
    [
      "/usr/lib64/python2.7/subprocess.py",
      711,
      "__init__",
      "errread, errwrite)"
    ],
    [
      "/usr/lib64/python2.7/subprocess.py",
      1343,
      "_execute_child",
      "raise child_exception"
    ]
  ],
  "errorType": "OSError",
  "errorMessage": "[Errno 2] No such file or directory"
}

Request ID:
"098c848f-87e7-45dd-89fd-4b79d0767907"

Function Logs:
START RequestId: 098c848f-87e7-45dd-89fd-4b79d0767907 Version: $LATEST
===[environment-variables]===
_HANDLER
AWS_LAMBDA_FUNCTION_VERSION
LAMBDA_TASK_ROOT
PATH
LD_LIBRARY_PATH
LANG
TZ
AWS_REGION
AWS_XRAY_CONTEXT_MISSING
AWS_SESSION_TOKEN
AWS_SECURITY_TOKEN
LAMBDA_RUNTIME_DIR
PYTHONPATH
AWS_LAMBDA_FUNCTION_MEMORY_SIZE
_AWS_XRAY_DAEMON_PORT
_AWS_XRAY_DAEMON_ADDRESS
AWS_LAMBDA_LOG_GROUP_NAME
AWS_LAMBDA_LOG_STREAM_NAME
AWS_ACCESS_KEY_ID
_X_AMZN_TRACE_ID
AWS_DEFAULT_REGION
AWS_SECRET_ACCESS_KEY
AWS_EXECUTION_ENV
AWS_XRAY_DAEMON_ADDRESS
AWS_LAMBDA_FUNCTION_NAME

===[runtime directory:/var/runtime]===
liblambdaruntime.so
docutils-0.14.dist-info
botocore-1.10.74.dist-info
futures-3.2.0.dist-info
s3transfer-0.1.13.dist-info
six-1.11.0.dist-info
boto3-1.7.74.dist-info
s3transfer
botocore
liblambdalog.so
dateutil
six.pyc
awslambda
bin
docutils
six.py
boto3
liblambdaio.so
jmespath
concurrent
python_dateutil-2.7.3.dist-info
jmespath-0.9.3.dist-info
liblambdaipc.so

===[amazon linux version]===
[Errno 2] No such file or directory: OSError
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 24, in lambda_handler
    ret = subprocess.call(cmd)
  File "/usr/lib64/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

END RequestId: 098c848f-87e7-45dd-89fd-4b79d0767907
REPORT RequestId: 098c848f-87e7-45dd-89fd-4b79d0767907  Duration: 93.64 ms  Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 42 MB

100

101

102

103

104

Response:

{

"stackTrace": [

[

"/var/task/lambda_function.py",

24,

"lambda_handler",

"ret = subprocess.call(cmd)"

[

"/usr/lib64/python2.7/subprocess.py",

523,

"call",

"return Popen(*popenargs, **kwargs).wait()"

[

"/usr/lib64/python2.7/subprocess.py",

711,

"__init__",

"errread, errwrite)"

[

"/usr/lib64/python2.7/subprocess.py",

1343,

"_execute_child",

"raise child_exception"

]

"errorType": "OSError",

"errorMessage": "[Errno 2] No such file or directory"

}

Request ID:

"098c848f-87e7-45dd-89fd-4b79d0767907"

Function Logs:

START RequestId: 098c848f-87e7-45dd-89fd-4b79d0767907 Version: $LATEST

===[environment-variables]===

_HANDLER

AWS_LAMBDA_FUNCTION_VERSION

LAMBDA_TASK_ROOT

PATH

LD_LIBRARY_PATH

LANG

AWS_REGION

AWS_XRAY_CONTEXT_MISSING

AWS_SESSION_TOKEN

AWS_SECURITY_TOKEN

LAMBDA_RUNTIME_DIR

PYTHONPATH

AWS_LAMBDA_FUNCTION_MEMORY_SIZE

_AWS_XRAY_DAEMON_PORT

_AWS_XRAY_DAEMON_ADDRESS

AWS_LAMBDA_LOG_GROUP_NAME

AWS_LAMBDA_LOG_STREAM_NAME

AWS_ACCESS_KEY_ID

_X_AMZN_TRACE_ID

AWS_DEFAULT_REGION

AWS_SECRET_ACCESS_KEY

AWS_EXECUTION_ENV

AWS_XRAY_DAEMON_ADDRESS

AWS_LAMBDA_FUNCTION_NAME

===[runtime directory:/var/runtime]===

liblambdaruntime.so

docutils-0.14.dist-info

botocore-1.10.74.dist-info

futures-3.2.0.dist-info

s3transfer-0.1.13.dist-info

six-1.11.0.dist-info

boto3-1.7.74.dist-info

s3transfer

botocore

liblambdalog.so

dateutil

six.pyc

awslambda

bin

docutils

six.py

boto3

liblambdaio.so

jmespath

concurrent

python_dateutil-2.7.3.dist-info

jmespath-0.9.3.dist-info

liblambdaipc.so

===[amazon linux version]===

[Errno 2] No such file or directory: OSError

Traceback (most recent call last):

File "/var/task/lambda_function.py", line 24, in lambda_handler

ret = subprocess.call(cmd)

File "/usr/lib64/python2.7/subprocess.py", line 523, in call

return Popen(*popenargs, **kwargs).wait()

File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__

errread, errwrite)

File "/usr/lib64/python2.7/subprocess.py", line 1343, in _execute_child

raise child_exception

OSError: [Errno 2] No such file or directory

END RequestId: 098c848f-87e7-45dd-89fd-4b79d0767907

REPORT RequestId: 098c848f-87e7-45dd-89fd-4b79d0767907 Duration: 93.64 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 42 MB

2.7でもboto3-1.7.74.dist-info

Lambda pythonバージョン

https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/python-programming-model.html

boto3 s3 署名バージョン対応

1.5.71 (Botocore)、1.4.6 (Boto3) にアップグレード。

【超重要】対応しないと使えなくなるかも？！今、全S3ユーザがチェックすべき署名バージョン２の廃止について

pythonコード

# -*- coding: utf-8 -*-
import os
import subprocess

def lambda_handler(event, context):

    # 環境変数一覧を取得
    print("===[environment-variables]===")
    for env in os.environ:
        print(env)

    #ランタイムの格納されているディレクトリを確認
    rtdir = os.environ.get("LAMBDA_RUNTIME_DIR")
    print("\n===[runtime directory:"+rtdir+"]===")
    
    #ディレクトリ内を表示
    files = os.listdir(rtdir)
    for file in files:
        print(file)

    # amazon linux version
    print("\n===[amazon linux version]===")
    cmd = "cat /etc/system-release";
    ret = subprocess.call(cmd)
    print(ret)

    cmd = "ls -la /etc/system*";
    ret = subprocess.call(cmd)
    print(ret)

    return 0

# -*- coding: utf-8 -*-

import os

import subprocess

def lambda_handler(event, context):

# 環境変数一覧を取得

print("===[environment-variables]===")

for env in os.environ:

print(env)

#ランタイムの格納されているディレクトリを確認

rtdir = os.environ.get("LAMBDA_RUNTIME_DIR")

print("\n===[runtime directory:"+rtdir+"]===")

#ディレクトリ内を表示

files = os.listdir(rtdir)

for file in files:

print(file)

# amazon linux version

print("\n===[amazon linux version]===")

cmd = "cat /etc/system-release";

ret = subprocess.call(cmd)

print(ret)

cmd = "ls -la /etc/system*";

ret = subprocess.call(cmd)

print(ret)

return 0

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/UsingAWSSDK.html

すべての AWS リージョンでは、AWS SDK はデフォルトで署名バージョン 4 を使用してリクエストを認証します。2016 年 5 月以前にリリースされた AWS SDK を使用する場合、次の表に示すように、署名バージョン 4 のリクエストが必要になることがあります。

[s3] use-sigv4 = True

5af85bf0

AWS SDK for PHPでのslab増大とNSS_SDB_USE_CACHE対策の効果確認

コメントを残す

色々調べると、GuzzleHttpがHTTPSで通信する際に増えるらしい

まとめ

AWS SDKのクライアント、GuzzleHttpも最終的にphpのcurl_～関数を呼んでいる
※constructorでcurlが無い場合、Exception吐くコードがあるし、AWSの要件にもcurlが入っている
php curl の nssは多分、ビルドされたcurlのNSSバージョンに依存してそう
※curl -V のNSS/3.27 の部分
curlをmakeすると、その時のOSのnssバージョンが組み込まれた
curl 7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.36
かつ、その際、nss-softokn-3.14.3-23.3.el6_8.x86_64 で、curlコマンドのNSS_SDB_USE_CACHE=yes は有効だった
phpのcurlバージョンは、コマンドと同じくlibcurlに依存してそうだが、curl_version(); 又は、phpinfo(); で確認できる

各種情報

[root@ip-172-31-16-82 centos]# cat /etc/redhat-release
CentOS release 6.9 (Final)
[root@ip-172-31-16-82 centos]# rpm -qa | grep nss-softokn
nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
nss-softokn-3.14.3-23.3.el6_8.x86_64
[root@ip-172-31-16-82 centos]# php -v
PHP 5.5.38 (cli) (built: Mar  1 2018 11:12:35)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2015 Zend Technologies


[centos@ip-172-31-16-82 slab-nss]$ cat composer.json
{
    "require": {
        "aws/aws-sdk-php": "^2.8"
    }
}

[root@ip-172-31-16-82 centos]# cat /etc/redhat-release

CentOS release 6.9 (Final)

[root@ip-172-31-16-82 centos]# rpm -qa | grep nss-softokn

nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64

nss-softokn-3.14.3-23.3.el6_8.x86_64

[root@ip-172-31-16-82 centos]# php -v

PHP 5.5.38 (cli) (built: Mar 1 2018 11:12:35)

[centos@ip-172-31-16-82 slab-nss]$ cat composer.json

{

"require": {

"aws/aws-sdk-php": "^2.8"

}

AWS-SDKでSNSへpublishするテストコードを準備

<?php

require_once('./vendor/autoload.php');
use Aws\Sns\SnsClient;

$arn = "arn:aws:sns:ap-northeast-1:xxxxxx:aws-sdk-nss-dentry-test";
$message = 'dummy message';

$client = SnsClient::factory(array(
    'profile' => 'default',
    'region'  => 'ap-northeast-1',
));

for ($i = 0; $i < 100; $i ++) {
    $result = $client->publish(array(
        'TopicArn' => $arn,
        'Message' => $message
    ));
}

<?php

require_once('./vendor/autoload.php');

use Aws\Sns\SnsClient;

$arn = "arn:aws:sns:ap-northeast-1:xxxxxx:aws-sdk-nss-dentry-test";

$message = 'dummy message';

$client = SnsClient::factory(array(

'profile' => 'default',

'region' => 'ap-northeast-1',

));

for ($i = 0; $i < 100; $i ++) {

$result = $client->publish(array(

'TopicArn' => $arn,

'Message' => $message

));

}

実行前後の情報

[centos@ip-172-31-16-82 ~]$ sar -v 0
Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal)      07/08/2018      _x86_64_        (1 CPU)

07:51:10 AM dentunusd   file-nr  inode-nr    pty-nr
07:51:10 AM     41344       736      7227         5
[centos@ip-172-31-16-82 ~]$ sar -v 0
Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal)      07/08/2018      _x86_64_        (1 CPU)

07:51:33 AM dentunusd   file-nr  inode-nr    pty-nr
07:51:33 AM     61344       736      7227         5

[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry
 48140  48105  99%    0.19K   2407       20      9628K dentry
[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry
 68100  68100 100%    0.19K   3405       20     13620K dentry

[centos@ip-172-31-16-82 ~]$ free
             total       used       free     shared    buffers     cached
Mem:        499996     174200     325796        160      14212      53744
-/+ buffers/cache:     106244     393752
Swap:      1048572       1280    1047292
[centos@ip-172-31-16-82 ~]$ free
             total       used       free     shared    buffers     cached
Mem:        499996     178176     321820        160      14220      53744
-/+ buffers/cache:     110212     389784
Swap:      1048572       1280    1047292

[centos@ip-172-31-16-82 ~]$ sar -v 0

Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal) 07/08/2018 _x86_64_ (1 CPU)

07:51:10 AM dentunusd file-nr inode-nr pty-nr

07:51:10 AM 41344 736 7227 5

[centos@ip-172-31-16-82 ~]$ sar -v 0

Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal) 07/08/2018 _x86_64_ (1 CPU)

07:51:33 AM dentunusd file-nr inode-nr pty-nr

07:51:33 AM 61344 736 7227 5

[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry

48140 48105 99% 0.19K 2407 20 9628K dentry

[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry

68100 68100 100% 0.19K 3405 20 13620K dentry

[centos@ip-172-31-16-82 ~]$ free

total used free shared buffers cached

Mem: 499996 174200 325796 160 14212 53744

-/+ buffers/cache: 106244 393752

Swap: 1048572 1280 1047292

[centos@ip-172-31-16-82 ~]$ free

total used free shared buffers cached

Mem: 499996 178176 321820 160 14220 53744

-/+ buffers/cache: 110212 389784

Swap: 1048572 1280 1047292

dentunusedが増加し、dentryのslabが増加している。

buffers/cacheが減少している。

対策コードを追加

<?php
putenv("NSS_SDB_USE_CACHE=yes"); // ★追加

require_once('./vendor/autoload.php');
use Aws\Sns\SnsClient;

$arn = "arn:aws:sns:ap-northeast-1:xxxxxx:aws-sdk-nss-dentry-test";
$message = 'dummy message';

$client = SnsClient::factory(array(
    'profile' => 'default',
    'region'  => 'ap-northeast-1',
));

for ($i = 0; $i < 100; $i ++) {
    $result = $client->publish(array(
        'TopicArn' => $arn,
        'Message' => $message
    ));
}

<?php

putenv("NSS_SDB_USE_CACHE=yes"); // ★追加

require_once('./vendor/autoload.php');

use Aws\Sns\SnsClient;

$arn = "arn:aws:sns:ap-northeast-1:xxxxxx:aws-sdk-nss-dentry-test";

$message = 'dummy message';

$client = SnsClient::factory(array(

'profile' => 'default',

'region' => 'ap-northeast-1',

));

for ($i = 0; $i < 100; $i ++) {

$result = $client->publish(array(

'TopicArn' => $arn,

'Message' => $message

));

}

実行前後の情報

[centos@ip-172-31-16-82 ~]$ sar -v 0
Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal)      07/08/2018      _x86_64_        (1 CPU)

07:55:05 AM dentunusd   file-nr  inode-nr    pty-nr
07:55:05 AM     61349       736      7230         5
[centos@ip-172-31-16-82 ~]$ sar -v 0
Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal)      07/08/2018      _x86_64_        (1 CPU)

07:55:15 AM dentunusd   file-nr  inode-nr    pty-nr
07:55:15 AM     61349       736      7230         5

[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry
 68120  68110  99%    0.19K   3406       20     13624K dentry
[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry
 68160  68136  99%    0.19K   3408       20     13632K dentry

[centos@ip-172-31-16-82 ~]$ free
             total       used       free     shared    buffers     cached
Mem:        499996     178300     321696        160      14292      53844
-/+ buffers/cache:     110164     389832
Swap:      1048572       1280    1047292
[centos@ip-172-31-16-82 ~]$ free
             total       used       free     shared    buffers     cached
Mem:        499996     178300     321696        160      14300      53844
-/+ buffers/cache:     110156     389840
Swap:      1048572       1280    1047292

[centos@ip-172-31-16-82 ~]$ sar -v 0

Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal) 07/08/2018 _x86_64_ (1 CPU)

07:55:05 AM dentunusd file-nr inode-nr pty-nr

07:55:05 AM 61349 736 7230 5

[centos@ip-172-31-16-82 ~]$ sar -v 0

Linux 2.6.32-696.30.1.el6.x86_64 (ip-172-31-16-82.ap-northeast-1.compute.internal) 07/08/2018 _x86_64_ (1 CPU)

07:55:15 AM dentunusd file-nr inode-nr pty-nr

07:55:15 AM 61349 736 7230 5

[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry

68120 68110 99% 0.19K 3406 20 13624K dentry

[centos@ip-172-31-16-82 ~]$ slabtop -o | grep dentry

68160 68136 99% 0.19K 3408 20 13632K dentry

[centos@ip-172-31-16-82 ~]$ free

total used free shared buffers cached

Mem: 499996 178300 321696 160 14292 53844

-/+ buffers/cache: 110164 389832

Swap: 1048572 1280 1047292

[centos@ip-172-31-16-82 ~]$ free

total used free shared buffers cached

Mem: 499996 178300 321696 160 14300 53844

-/+ buffers/cache: 110156 389840

Swap: 1048572 1280 1047292

ほぼ増加しなくなったことがわかる。

nss-softokn-3.14.3-23.3.el6_8 の環境でも効果がありそうだ。

GuzzleHttp

コードを追っていくと、内部でphp curlを使っているようだ

if (!extension_loaded('curl')) {
    // @codeCoverageIgnoreStart
    throw new RuntimeException('The PHP cURL extension must be installed to use Guzzle.');
    // @codeCoverageIgnoreEnd
}

if (!extension_loaded('curl')) {

// @codeCoverageIgnoreStart

throw new RuntimeException('The PHP cURL extension must be installed to use Guzzle.');

// @codeCoverageIgnoreEnd

}

curl と nss

[centos@ip-172-31-16-82 slab-nss]$ php -r "phpinfo();" | grep "^curl" -A24
PHP Warning:  phpinfo(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in Command line code on line 1
curl

cURL support => enabled
cURL Information => 7.19.7
Age => 3
Features
AsynchDNS => No
CharConv => No
Debug => No
GSS-Negotiate => Yes
IDN => Yes
IPv6 => Yes
krb4 => No
Largefile => Yes
libz => Yes
NTLM => Yes
SPNEGO => No
SSL => Yes
SSPI => No
Protocols => tftp, ftp, telnet, dict, ldap, ldaps, http, file, https, ftps, scp, sftp
Host => x86_64-redhat-linux-gnu
SSL Version => NSS/3.27.1
ZLib Version => 1.2.3
libSSH Version => libssh2/1.4.2

[centos@ip-172-31-16-82 slab-nss]$ php -r "phpinfo();" | grep "^curl" -A24

PHP Warning: phpinfo(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in Command line code on line 1

curl

cURL support => enabled

cURL Information => 7.19.7

Age => 3

Features

AsynchDNS => No

CharConv => No

Debug => No

GSS-Negotiate => Yes

IDN => Yes

IPv6 => Yes

krb4 => No

Largefile => Yes

libz => Yes

NTLM => Yes

SPNEGO => No

SSL => Yes

SSPI => No

Protocols => tftp, ftp, telnet, dict, ldap, ldaps, http, file, https, ftps, scp, sftp

Host => x86_64-redhat-linux-gnu

SSL Version => NSS/3.27.1

ZLib Version => 1.2.3

libSSH Version => libssh2/1.4.2

[centos@ip-172-31-16-82 slab-nss]$ php -r "print_r(curl_version());"
Array
(
    [version_number] => 463623
    [age] => 3
    [features] => 1597
    [ssl_version_number] => 0
    [version] => 7.19.7
    [host] => x86_64-redhat-linux-gnu
    [ssl_version] => NSS/3.27.1
    [libz_version] => 1.2.3
    [protocols] => Array
        (
            [0] => tftp
            [1] => ftp
            [2] => telnet
            [3] => dict
            [4] => ldap
            [5] => ldaps
            [6] => http
            [7] => file
            [8] => https
            [9] => ftps
            [10] => scp
            [11] => sftp
        )

)

[centos@ip-172-31-16-82 slab-nss]$ php -r "print_r(curl_version());"

Array

(

[version_number] => 463623

[age] => 3

[features] => 1597

[ssl_version_number] => 0

[version] => 7.19.7

[host] => x86_64-redhat-linux-gnu

[ssl_version] => NSS/3.27.1

[libz_version] => 1.2.3

[protocols] => Array

(

[0] => tftp

[1] => ftp

[2] => telnet

[3] => dict

[4] => ldap

[5] => ldaps

[6] => http

[7] => file

[8] => https

[9] => ftps

[10] => scp

[11] => sftp

)

NSS/3.27.1 が使われているようだ。

osにinstallされているnss-softknは使われていない？

curlをmakeして確かめてみる

同じバージョンをインストール

#  cd /usr/local/src/
#  curl -O https://curl.haxx.se/download/archeology/curl-7.19.7.tar.gz
#  tar xvzf curl-7.19.7.tar.gz
#  cd curl-7.19.7

#  yum install gcc g++
#  yum install nspr-devel
#  yum install nss-devel

#  ./configure --with-nss --disable-manual
#  make

# cd /usr/local/src/

# curl -O https://curl.haxx.se/download/archeology/curl-7.19.7.tar.gz

# tar xvzf curl-7.19.7.tar.gz

# cd curl-7.19.7

# yum install gcc g++

# yum install nspr-devel

# yum install nss-devel

# ./configure --with-nss --disable-manual

# make

システムのnssが使われた事の確認と効果の確認

[root@ip-172-31-16-82 curl-7.19.7]# rpm -qa | grep nss
nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
nss-util-3.36.0-1.el6.x86_64
nss-softokn-freebl-devel-3.14.3-23.3.el6_8.x86_64
nss-sysinit-3.36.0-8.el6.x86_64
nss-devel-3.36.0-8.el6.x86_64
openssh-server-5.3p1-123.el6_9.x86_64
nss-util-devel-3.36.0-1.el6.x86_64
nss-softokn-devel-3.14.3-23.3.el6_8.x86_64
nss-3.36.0-8.el6.x86_64
nss-tools-3.36.0-8.el6.x86_64
nss-softokn-3.14.3-23.3.el6_8.x86_64
openssl-1.0.1e-57.el6.x86_64
openssh-5.3p1-123.el6_9.x86_64
openssh-clients-5.3p1-123.el6_9.x86_64

[root@ip-172-31-16-82 curl-7.19.7]# ./src/curl -V
curl 7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.36
Protocols: tftp ftp telnet dict http file https ftps
Features: IPv6 Largefile SSL

[root@ip-172-31-16-82 curl-7.19.7]# strace -fc -e trace=access ./src/curl  -s https://www.google.co.jp/ > /dev/null
Process 28940 attached
Process 28941 attached
Process 28942 attached
Process 28943 attached
Process 28944 attached
Process 28946 attached
Process 28945 attached
Process 28947 attached
Process 28949 attached
Process 28948 attached
Process 28950 attached
Process 28951 attached
Process 28953 attached
Process 28952 attached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0       938       926 access
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   938       926 total
[root@ip-172-31-16-82 curl-7.19.7]# NSS_SDB_USE_CACHE=yes strace -fc -e trace=access ./src/curl  -s https://www.google.co.jp/ > /dev/null
Process 29010 attached
Process 29011 attached
Process 29012 attached
Process 29014 attached
Process 29015 attached
Process 29017 attached
Process 29016 attached
Process 29018 attached
Process 29020 attached
Process 29019 attached
Process 29021 attached
Process 29022 attached
Process 29024 attached
Process 29023 attached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0        24        12 access
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                    24        12 total

[root@ip-172-31-16-82 curl-7.19.7]# rpm -qa | grep nss

nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64

nss-util-3.36.0-1.el6.x86_64

nss-softokn-freebl-devel-3.14.3-23.3.el6_8.x86_64

nss-sysinit-3.36.0-8.el6.x86_64

nss-devel-3.36.0-8.el6.x86_64

openssh-server-5.3p1-123.el6_9.x86_64

nss-util-devel-3.36.0-1.el6.x86_64

nss-softokn-devel-3.14.3-23.3.el6_8.x86_64

nss-3.36.0-8.el6.x86_64

nss-tools-3.36.0-8.el6.x86_64

nss-softokn-3.14.3-23.3.el6_8.x86_64

openssl-1.0.1e-57.el6.x86_64

openssh-5.3p1-123.el6_9.x86_64

openssh-clients-5.3p1-123.el6_9.x86_64

[root@ip-172-31-16-82 curl-7.19.7]# ./src/curl -V

curl 7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.36

Protocols: tftp ftp telnet dict http file https ftps

Features: IPv6 Largefile SSL

[root@ip-172-31-16-82 curl-7.19.7]# strace -fc -e trace=access ./src/curl -s https://www.google.co.jp/ > /dev/null

Process 28940 attached

Process 28941 attached

Process 28942 attached

Process 28943 attached

Process 28944 attached

Process 28946 attached

Process 28945 attached

Process 28947 attached

Process 28949 attached

Process 28948 attached

Process 28950 attached

Process 28951 attached

Process 28953 attached

Process 28952 attached

% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ----------------

0.00 0.000000 0 938 926 access

------ ----------- ----------- --------- --------- ----------------

100.00 0.000000 938 926 total

[root@ip-172-31-16-82 curl-7.19.7]# NSS_SDB_USE_CACHE=yes strace -fc -e trace=access ./src/curl -s https://www.google.co.jp/ > /dev/null

Process 29010 attached

Process 29011 attached

Process 29012 attached

Process 29014 attached

Process 29015 attached

Process 29017 attached

Process 29016 attached

Process 29018 attached

Process 29020 attached

Process 29019 attached

Process 29021 attached

Process 29022 attached

Process 29024 attached

Process 29023 attached

% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ----------------

0.00 0.000000 0 24 12 access

------ ----------- ----------- --------- --------- ----------------

100.00 0.000000 24 12 total

nss-softokn-3.14.3-23.3 で nss-softokn-3.14.3-23.3 が使えているだろうと思われる。

AWSのt2.microでAWS SKDのcomposerがこけたので、スワップを作成

コメントを残す

[centos@ip-172-31-16-82 slab-nss]$ composer.phar update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 3 installs, 0 updates, 0 removals
  - Installing symfony/event-dispatcher (v2.6.13): Downloading (100%)
The following exception is caused by a lack of memory or swap, or not having swap configured
Check https://getcomposer.org/doc/articles/troubleshooting.md#proc-open-fork-failed-errors for details

PHP Warning:  proc_open(): fork failed - Cannot allocate memory in phar:///usr/local/bin/composer.phar/vendor/symfony/console/Application.php on line 959

Warning: proc_open(): fork failed - Cannot allocate memory in phar:///usr/local/bin/composer.phar/vendor/symfony/console/Application.php on line 959

  [ErrorException]
  proc_open(): fork failed - Cannot allocate memory


update [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--lock] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [--with-dependencies] [--with-all-dependencies] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [-i|--interactive] [--root-reqs] [--] [<packages>]...

[centos@ip-172-31-16-82 slab-nss]$ composer.phar update

Loading composer repositories with package information

Updating dependencies (including require-dev)

Package operations: 3 installs, 0 updates, 0 removals

- Installing symfony/event-dispatcher (v2.6.13): Downloading (100%)

The following exception is caused by a lack of memory or swap, or not having swap configured

Check https://getcomposer.org/doc/articles/troubleshooting.md#proc-open-fork-failed-errors for details

PHP Warning: proc_open(): fork failed - Cannot allocate memory in phar:///usr/local/bin/composer.phar/vendor/symfony/console/Application.php on line 959

Warning: proc_open(): fork failed - Cannot allocate memory in phar:///usr/local/bin/composer.phar/vendor/symfony/console/Application.php on line 959

[ErrorException]

proc_open(): fork failed - Cannot allocate memory

update [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--lock] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [--with-dependencies] [--with-all-dependencies] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [-i|--interactive] [--root-reqs] [--] [<packages>]...

となったので、

[root@ip-172-31-16-82 slab-nss]# /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 15.1272 s, 71.0 MB/s
[root@ip-172-31-16-82 slab-nss]# /sbin/mkswap /var/swap.1
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=001c4bd6-1612-4241-9a0c-8fe3b27f5e08
[root@ip-172-31-16-82 slab-nss]# /sbin/swapon /var/swap.1

[root@ip-172-31-16-82 slab-nss]# /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 15.1272 s, 71.0 MB/s

[root@ip-172-31-16-82 slab-nss]# /sbin/mkswap /var/swap.1

Setting up swapspace version 1, size = 1048572 KiB

no label, UUID=001c4bd6-1612-4241-9a0c-8fe3b27f5e08

[root@ip-172-31-16-82 slab-nss]# /sbin/swapon /var/swap.1

と作成

[centos@ip-172-31-16-82 slab-nss]$ composer.phar update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 3 installs, 0 updates, 0 removals
  - Installing symfony/event-dispatcher (v2.6.13): Loading from cache
  - Installing guzzle/guzzle (v3.9.3): Downloading (100%)
  - Installing aws/aws-sdk-php (2.8.31): Downloading (100%)
symfony/event-dispatcher suggests installing symfony/dependency-injection ()
symfony/event-dispatcher suggests installing symfony/http-kernel ()
guzzle/guzzle suggests installing guzzlehttp/guzzle (Guzzle 5 has moved to a new package name. The package you have installed, Guzzle 3, is deprecated.)
aws/aws-sdk-php suggests installing doctrine/cache (Adds support for caching of credentials and responses)
aws/aws-sdk-php suggests installing ext-apc (Allows service description opcode caching, request and response caching, and credentials caching)
aws/aws-sdk-php suggests installing monolog/monolog (Adds support for logging HTTP requests and responses)
aws/aws-sdk-php suggests installing symfony/yaml (Eases the ability to write manifests for creating jobs in AWS Import/Export)
Package guzzle/guzzle is abandoned, you should avoid using it. Use guzzlehttp/guzzle instead.
Writing lock file
Generating autoload files

[centos@ip-172-31-16-82 slab-nss]$ composer.phar update

Loading composer repositories with package information

Updating dependencies (including require-dev)

Package operations: 3 installs, 0 updates, 0 removals

- Installing symfony/event-dispatcher (v2.6.13): Loading from cache

- Installing guzzle/guzzle (v3.9.3): Downloading (100%)

- Installing aws/aws-sdk-php (2.8.31): Downloading (100%)

symfony/event-dispatcher suggests installing symfony/dependency-injection ()

symfony/event-dispatcher suggests installing symfony/http-kernel ()

guzzle/guzzle suggests installing guzzlehttp/guzzle (Guzzle 5 has moved to a new package name. The package you have installed, Guzzle 3, is deprecated.)

aws/aws-sdk-php suggests installing doctrine/cache (Adds support for caching of credentials and responses)

aws/aws-sdk-php suggests installing ext-apc (Allows service description opcode caching, request and response caching, and credentials caching)

aws/aws-sdk-php suggests installing monolog/monolog (Adds support for logging HTTP requests and responses)

aws/aws-sdk-php suggests installing symfony/yaml (Eases the ability to write manifests for creating jobs in AWS Import/Export)

Package guzzle/guzzle is abandoned, you should avoid using it. Use guzzlehttp/guzzle instead.

Writing lock file

Generating autoload files

とできた。

furoom

カテゴリーアーカイブ:AWS SDK

Lambda Python runtime environment

AWS SDK for PHPでのslab増大とNSS_SDB_USE_CACHE対策の効果確認

まとめ

各種情報

AWS-SDKでSNSへpublishするテストコードを準備

実行前後の情報

対策コードを追加

実行前後の情報

GuzzleHttp

curl と nss

curlをmakeして確かめてみる

AWSのt2.microでAWS SKDのcomposerがこけたので、スワップを作成