Issue : server closed the connection unexpectedly This probably means the server terminated abnormally

[adnanhamdussalam]

Hi Team,

I am facing against the below query as per my logs:

This probably means the server terminated abnormally
    before or while processing the request.",,,,,,"/* {""app"": ""dbt"", ""dbt_version"": ""10.2"", ""profile_name"": ""rio-dev-profile"", ""target_name"": ""dev"", ""node_id"": ""model.riodbt.ca""} */

create temporary table ""aca__dbt_tmp120645746261""

as

(

WITH base_pre_aca_filtered AS (
    SELECT *
    FROM ""gpmyafi"".""my_28"".""pre_aca"" aca


            WHERE aca.call_seg_start_time >= '2026-04-20 21:56:12'::DATE - interval '90 days'


),
aca_join_agent_attribute AS (
    SELECT
        aca.*
        , aa.agent_skilling
    FROM my_pre_a_filtered_base aca
    LEFT JOIN ""gpmyafi"".""my_28"".""t_agent_attributes"" as aa
    ON aca.ans_agent_id = aa.agent_id
    AND (aca.call_seg_start_time >= aa.login_time AND (aca.call_seg_start_time < aa.logout_time OR aa.logout_time is NULL))
),

aca_join_haglog_1 AS (
    SELECT
        aca.primary_telephony_id
        , array_to_string(array_agg( distinct concat( h.skill_name, ':', h.skill_level::int) order by concat( h.skill_name, ':', h.skill_level::int) ) , ',') as agent_skilling
    FROM my_pre_a_filtered_base aca
    LEFT JOIN ""gpmyafi"".""my_28"".""haglog"" as h
    ON aca.switch_agent_id = h.agent_skill_employee_id::text
    AND ( aca.call_seg_start_time >= h.agent_skill_start_ts AND aca.call_seg_start_time < h.agent_skill_end_ts )
    where h.skill_name is not null and h.skill_level is not null
    GROUP BY aca.primary_telephony_id
),

aca_join_haglog_2 AS (
    SELECT
        aca.primary_telephony_id
        , array_to_string(array_agg(distinct concat(h.agent_group, ':', -1::int) order by concat(h.agent_group, ':', -1::int)) , ',') as agent_skilling
    FROM my_pre_a_filtered_base aca
    LEFT JOIN ""gpmyafi"".""my_28"".""haglog"" as h
    ON aca.switch_agent_id = h.agent_group_employee_id::text
    AND ( aca.call_seg_start_time >= h.agent_group_start_ts AND aca.call_seg_start_time < h.agent_group_end_ts )
    where h.agent_group is not null
    GROUP BY aca.primary_telephony_id
),

aca_join_haglog_3 AS
(
    SELECT
        coalesce(h1.primary_telephony_id,h2.primary_telephony_id) as primary_telephony_id
        , concat_my(',',h1.agent_skilling, h2.agent_skilling) as agent_skilling
    FROM aca_join_haglog_1 h1
    full outer JOIN aca_join_haglog_2 as h2
    ON h1.primary_telephony_id = h2.primary_telephony_id
),

aca_join_haglog AS
(
    SELECT
        aca.primary_telephony_id,
        aca.call_id,
        aca.call_date,
        aca.call_seg_start_time,
        aca.call_seg_start_time_utc,
        aca.call_answer_time,
        aca.call_hang_up_time,
        aca.call_seg_end_time,
        aca.next_call_answer_time,
        aca.first_call_answer_time,
        aca.skill_id,
        aca.vdn_id,
        aca.routing_scope_id,
        aca.switch_agent_id,
        aca.call_disp,
        aca.transferred,
        aca.acd_attached_data,
        aca.first_telephony_id,
        aca.customer_phone_number,
        aca.customer_id_1,
        aca.customer_id_2,
        aca.customer_id_3,
        aca.customer_id_4,
        aca.customer_id_5,
        aca.customer_id_6,
        aca.parsed_customer_id,
        aca.parsed_phone_number,
        aca.native_skill_data,
        aca.target_expression,
        aca.routing_status,
        aca.acd_routing_status,
        aca.handle_time,
        aca.on_off,
        aca.extended_benchmark,
        aca.ans_agent_wait_time,
        aca.abandon,
        aca.outcome_id,
        aca.outcome_key,
        aca.outcome_status,
        aca.outcome_type,
        aca.outcome_callid,
        aca.outcome_customer_id_1,
        aca.outcome_customer_id_2,
        aca.outcome_customer_id_3,
        aca.outcome_customer_id_4,
        aca.outcome_customer_id_5,
        aca.outcome_customer_id_6,
        aca.outcome_agent_id,
        aca.outcome_binary,
        aca.outcome_revenue,
        aca.outcome_categorical,
        aca.outcome_continuous,
        aca.outcome_create_time,
        aca.outcome_create_time_secondary,
        aca.outcome_confirmation,
        aca.outcome_confirmation_agent,
        aca.outcome_confirmation_time,
        aca.outcome_confirmation_days,
        aca.attribution_outcome_id,
        aca.attribution_outcome_key,
        aca.attribution_outcome_binary,
        aca.attribution_outcome_revenue,
        aca.attribution_outcome_continuous,
        aca.attribution_outcome_create_time,
        aca.attribution_outcome_create_time_secondary,
        aca.attribution_outcome_status,
        aca.attribution_outcome_mode,

        aca.outcome_details,
        aca.p_vdn,
        aca.o_vdn,
        aca.from_transfer,
        aca.dialed_num,
        aca.call_priority,
        aca.agent_priority,
        aca.agent_choice,
        aca.caller_choice,
        aca.agent_choice_used,
        aca.caller_choice_used,
        aca.agent_percentile_actual,
        aca.call_percentile_actual,
        aca.anslogin,
        aca.origlogin,
        aca.disppriority,
        aca.request_type,
        aca.target,
        aca.day0_target,
        aca.is_relevant,
        aca.is_relevant_switch,
        aca.is_relevant_final,
        guid,
        aca.acd_error_code,
        aca.agent_station,
            default_route_reason,
        aca.agent_group,
        aca.agent_group_string,
        aca.interaction_reason,
        aca.routed_agent_group,
        aca.ans_agent_id,
        aca.call_direction,
        coalesce(aca.agent_skilling, h.agent_skilling) agent_skilling







            ,

            outcome_id_19,
            outcome_binary_19,
            outcome_revenue_19,
            outcome_continuous_19

        , aca.ixn_id_outcomes
        , aca.agent_mapping_id
        , aca.queued_skill_list
        , aca.customer_benchmark



        , aca.call_guid
        , aca.original_leg

            , aca.original_interaction_id


    FROM my_a_j_agent_atrib aca
    LEFT JOIN aca_join_haglog_3 as h
    ON aca.primary_telephony_id = h.primary_telephony_id
)


SELECT *
    , left(aca.customer_phone_number,7) as customer_phone_number_grf
FROM aca_join_haglog aca

);

",0,,"cdbdispatchresult.c",510,"Stack trace:
1 0xd6a018 postgres errstart + 0x258
2 0xd6fd76 postgres ThrowErrorData + 0x16
3 0xa0e216 postgres mppExecutorFinishup + 0xd6
4 0x9ff1fa postgres standard_ExecutorEnd + 0xca
5 0x968fb8 postgres ExecCreateTableAs + 0x398
6 0xc1ef12 postgres + 0xc1ef12
7 0xc1dd49 postgres standard_ProcessUtility + 0x369
8 0xc1e893 postgres ProcessUtility + 0xb3
9 0xc1c2d5 postgres + 0xc1c2d5
10 0xc1c3fb postgres + 0xc1c3fb
11 0xc1ca03 postgres PortalRun + 0x2a3
12 0xc15ed3 postgres exec_simple_query + 0x1b3
13 0xc1a824 postgres PostgresMain + 0x2534
14 0xb78390 postgres + 0xb78390
15 0xb7939c postgres PostmasterMain + 0xe0c
16 0x7c45eb postgres main + 0x56b
17 0x7fd52e62a610 libc.so.6 + 0x2e62a610
18 0x7fd52e62a6c0 libc.so.6 __libc_start_main + 0x80
19 0x7d5075 postgres _start + 0x25
"

Kindly can you tell is it a bug?

[my-ship-it]

Hey @adnanhamdussalam, thanks for the report — and for including the stack, that's already a useful starting point.

I took a look, but I don't think we can call this a bug yet — at least not from what's here. The stack you pasted is actually from the coordinator side (mppExecutorFinishup → cdbdispatchresult.c:510). That's the path the QD goes down when it tries to collect results from a segment and finds the connection has already dropped. So the coordinator itself is fine — it's basically just telling us "one of my segments died on me." The real crash happened on a segment, and that part isn't in what you shared.

The upside: the stack does rule out a lot. It's not a parser/planner issue, not a cancellation or deadlock, not a resource-group admission thing, and not a QD bug. What's left is roughly:

• a segment getting OOM-killed (honestly my first guess given the 90-day window plus those array_agg(distinct concat(...) order by ...) aggregates),
• a segfault inside a QE — often a custom UDF, and concat_my jumps out at me here,
• or some PANIC path that fired after interconnect teardown (less likely).

To tell which one it is, would you mind grabbing a few extra things when you get a chance?

The two that would help the most:

• Segment logs around the time it failed. Something like `grep -E 'PANIC|SIGSEGV|signal|terminating' pg_log/cbdb-*.log` on the segment hosts — the PANIC/signal line and the stack right after it is exactly what's missing.
• `dmesg -T | grep -iE 'killed process|out of memory'` on the segment hosts. If the OOM-killer was involved, this is where it'll show up.

And if it's easy:

• The Cloudberry version (`SELECT version();`).
• The definition of `concat_my` (`\df+ concat_my`). If it's a custom C function, could you try swapping it for `concat_ws(',', h1.agent_skilling, h2.agent_skilling)` once and see if the crash still happens? That alone would tell us a lot.
• Does it fail every time, or only on certain data sizes? Does a smaller window (say 7 days instead of 90) still reproduce it?
• Values for `statement_mem`, `max_statement_mem`, `gp_workfile_limit_per_segment`, and which resource group this is running under.

A coredump from a segment would be ideal if you happen to have `kernel.core_pattern` set up, but no worries if not — even just the segment log line and `dmesg` output should let us classify this pretty quickly.

Thanks again, and looking forward to whatever you can share!

[adnanhamdussalam]

Hi @my-ship-it,

PFB the update:
The Cloudberry version (SELECT version();).
---- SELECT version();
version

PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating build bdf90c5) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit compiled on Mar 4 2026 23:25:31

The definition of concat_my (\df+ concat_my). If it's a custom C function, could you try swapping it for concat_ws(',', h1.agent_skilling, h2.agent_skilling) once and see if the crash still happens? That alone would tell us a lot.
-- it is concat_ws being used as function
Does it fail every time, or only on certain data sizes? Does a smaller window (say 7 days instead of 90) still reproduce it?
-- it is in progresss will update in a while
Values for statement_mem, max_statement_mem, gp_workfile_limit_per_segment, and which resource group this is running under.

gpconfig -s statement_mem
Values on all segments are consistent
GUC : statement_mem
Coordinator value: 256MB
Segment value: 256MB

gpconfig -s max_statement_mem
Values on all segments are consistent
GUC : max_statement_mem
Coordinator value: 10GB
Segment value: 10GB

gpconfig -s gp_workfile_limit_per_segment
Values on all segments are consistent
GUC : gp_workfile_limit_per_segment
Coordinator value: 0
Segment value: 0

Segment logs around the time it failed:

Found these errors from segment that is causing issue:
ERROR","25M01","canceling MPP operation"
"LOG","00000","Process interrupt for 'query cancel pending' (execScan.c:42)",
"LOG","00000","An exception was encountered during the execution of statement:

dmesg -T | grep -iE 'killed process|out of memory' :
No output on coordinator and segment nodes

[NadeemRajput-sys]

@my-ship-it it is observed that whenever CTE is executed that expect to process larger data set, such segment failure occures. If we convert CTE into using temp-tables instead, it mostly works fine.
CTE with smaller dataset processing works okie mostly.

Question is:

Is it a Bug or support limitation to CTE in Apache Cloudberry? same CTEs works fine in GP6.x

[adnanhamdussalam]

Hi @my-ship-it,

The query that I have shared earlier now crashed the segment:

PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1 0xd704a5 postgres StandardHandlerForSigillSigsegvSigbus_OnMainThread + 0x155

Kindly can you share a workaround about this issue.