Amazon redshift AWS红移:致命:超过非引导用户的连接限制“500”
希望你们都没事。 我们经常达到这个极限。我们知道在Redshift中无法达到500个并发用户连接的限制。我们还知道某些视图pg_user_info提供了有关用户实际限制的信息。 我们正在寻找一些在此论坛中找不到的答案,以及基于您经验的任何指导。 问题: 使用更大的EC2实例重新创建集群是否会产生更高的限制值? 向现有集群添加新节点是否会产生更高的限制值? 从应用程序开发的角度来看:为了发现或预测达到此极限的情况,您会推荐哪些具体策略/行动?Amazon redshift AWS红移:致命:超过非引导用户的连接限制“500”,amazon-redshift,amazon-redshift-spectrum,Amazon Redshift,Amazon Redshift Spectrum,希望你们都没事。 我们经常达到这个极限。我们知道在Redshift中无法达到500个并发用户连接的限制。我们还知道某些视图pg_user_info提供了有关用户实际限制的信息。 我们正在寻找一些在此论坛中找不到的答案,以及基于您经验的任何指导。 问题: 使用更大的EC2实例重新创建集群是否会产生更高的限制值? 向现有集群添加新节点是否会产生更高的限制值? 从应用程序开发的角度来看:为了发现或预测达到此极限的情况,您会推荐哪些具体策略/行动? Txs-Jimmy正如你所说,这是红移的一个硬限制,
Txs-Jimmy正如你所说,这是红移的一个硬限制,没有办法提高它。Redshift不是高并发/高连接数据库 我希望,如果您需要Redshift的强大数据分析能力,您可以通过连接共享解决这个问题。Pgpool是用于此目的的常用工具 好的,伙计们。 感谢所有回答的人。 我在AWS上贴了一张支持票,这是我的建议,粘贴在这里,虽然很长,但我希望它对很多遇到这个问题的人都有效。这样做的目的是在情况发生之前抓住它:
您的Amazon Redshift集群是否有500多名并发用户?这些是真人,还是正在连接的应用程序?你真的需要那么多的连接吗?他们都在干什么?
To monitor the number of connections made to the database, you can create a cloudwatch alarm based on the Database connections metrics that will trigger a lambda function when a certain threshold is reached. This lambda function can then terminate idle connections by calling a procedure that terminates idle connections.
Please find the query that creates a procedure to log and terminate long running inactive sessions
:
1. Add view to get all current inactive sessions in the cluster
CREATE OR REPLACE VIEW inactive_sessions as (
select a.process,
trim(a.user_name) as user_name,
trim(c.remotehost) as remotehost,
a.usesysid,
a.starttime,
datediff(s,a.starttime,sysdate) as session_dur,
b.last_end,
datediff(s,case when b.last_end is not null then b.last_end else a.starttime end,sysdate) idle_dur
FROM
(
select starttime,process,u.usesysid,user_name
from stv_sessions s, pg_user u
where
s.user_name = u.usename
and u.usesysid>1
and process NOT IN (select pid from stv_inflight where userid>1
union select pid from stv_recents where status != 'Done' and userid>1)
) a
LEFT OUTER JOIN (
select
userid,pid,max(endtime) as last_end from svl_statementtext
where userid>1 and sequence=0 group by 1,2) b ON a.usesysid = b.userid AND a.process = b.pid
LEFT OUTER JOIN (
select username, pid, remotehost from stl_connection_log
where event = 'initiating session' and username <> 'rsdb') c on a.user_name = c.username AND a.process = c.pid
WHERE (b.last_end > a.starttime OR b.last_end is null)
ORDER BY idle_dur
);
2. Add table for logging information about long running transactions that was terminated
CREATE TABLE IF NOT EXISTS terminated_inactive_sessions (
process int,
user_name varchar(50),
remotehost varchar(50),
starttime timestamp,
session_dur int,
idle_dur int,
terminated_on timestamp DEFAULT GETDATE()
);
3. Add procedure to log and terminate any inactive transactions running for longer than 'n' amount of seconds
CREATE OR REPLACE PROCEDURE terminate_and_log_inactive_sessions (n INTEGER)
AS $$
DECLARE
expired RECORD ;
BEGIN
FOR expired IN SELECT process, user_name, remotehost, starttime, session_dur, idle_dur FROM inactive_sessions where idle_dur >= n
LOOP
EXECUTE 'INSERT INTO terminated_inactive_sessions (process, user_name, remotehost, starttime, session_dur, idle_dur) values (' || expired.process || ' , ''' || expired.user_name || ''' , ''' || expired.remotehost || ''' , ''' || expired.starttime || ''' , ' || expired.session_dur || ' , ' || expired.idle_dur || ');';
EXECUTE 'SELECT PG_TERMINATE_BACKEND(' || expired.process || ')';
END LOOP ;
END ;
$$ LANGUAGE plpgsql;
4. Execute the procedure by running the following command:
call terminate_and_log_inactive_sessions(100);
Here is a sample lambda function that attempts to close idle connections by querying the view 'inactive_sessions' created above, which you can use as a reference.
#Current time
now = datetime.datetime.now()
query = "SELECT process, user_name, session_dur, idle_dur FROM inactive_sessions where idle_dur >= %d"
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
try:
conn = psycopg2.connect("dbname=" + db_database + " user=" + db_user + " password=" + db_password + " port=" + db_port + " host=" + db_host)
conn.autocommit = True
except:
logger.error("ERROR: Unexpected error: Could not connect to Redshift cluster.")
sys.exit()
logger.info("SUCCESS: Connection to RDS Redshift cluster succeeded")
with conn.cursor() as cur:
cur.execute(query % (session_idle_limit))
row_count = cur.rowcount
if row_count >=1:
result = cur.fetchall()
for row in result:
print("terminating session with pid %s that has been idle for %d seconds at %s" % (row[0],row[3],now))
cur.execute("SELECT PG_TERMINATE_BACKEND(%s);" % (row[0]))
conn.close()
else:
conn.close()