The bot metrics dashboard can show `handoff_rate + resolution_rate >
100%`. A single conversation can accumulate both
`conversation_bot_handoff` and `conversation_bot_resolved` events, and
the rate queries count them independently against a shared denominator.
## How it happens
```
Customer messages bot inbox
│
▼
┌──────────┐
│ pending │ (bot handling)
└────┬─────┘
│ bot can't help
▼
┌──────────┐
│ open │ (handed off → conversation_bot_handoff event created)
└────┬─────┘
│ agent clicks "Resolve" WITHOUT sending a message
▼
┌──────────┐
│ resolved │ conversation_resolved fires
└──────────┘
│
▼
create_bot_resolved_event guard checks:
✅ inbox.active_bot?
✅ no outgoing messages with sender_type: 'User' ← agent never messaged!
│
▼
conversation_bot_resolved event ALSO created ← BUG
│
▼
Same conversation counted in BOTH rates → sum exceeds 100%
```
## Why fix at the read path, not the write path
An earlier attempt added guards in the listener to make the two events
mutually exclusive per conversation — deleting `bot_resolved` when a
handoff fires, suppressing resolutions when a handoff exists. This was
rejected because conversations can be reopened across multiple cycles
(bot resolves on day 1, customer returns on day 5, bot hands off).
Deleting the day-1 resolution corrupts historical reports, and the async
event dispatcher makes listener-level guards vulnerable to race
conditions.
## What this PR does
Within a reporting window, if a conversation has both events, **handoff
wins** — the conversation is excluded from the resolution count. This is
applied via SQL subquery across all three read paths:
```
┌─────────────────────────┐
│ Reporting Events DB │
│ │
│ conv_bot_handoff: [A,B] │
│ conv_bot_resolved: [A,C]│
└────────┬────────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
BotMetricsBuilder ReportHelper CountReportBuilder
(rate cards) (bot_summary) (timeseries charts)
│ │ │
▼ ▼ ▼
resolutions: resolutions: resolutions:
[A,C] minus [A,B] same logic same logic
= [C] only = [C] only = [C] only
Result: Conversation A → handoff only
Conversation B → handoff only
Conversation C → resolution only
```
For wide date ranges spanning multiple lifecycles, a conversation
bot-resolved in one cycle and handed off in a later cycle will only show
as a handoff. This is an acceptable tradeoff — the alternative (>100%
rates) is clearly worse, and narrow ranges handle this correctly since
the events fall into different windows. No reporting events are
modified, so historical data stays intact.
## Diagnostic tool
`rake bot_metrics:diagnose` — read-only task that prompts for account ID
and date range, shows a before/after rate comparison without modifying
data.
---------
Co-authored-by: aakashb95 <aakashbakhle@gmail.com>
Co-authored-by: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com>
## PR2: Report builder refactor — DataSource abstraction
The existing report builders (timeseries + summary) had their SQL
queries inlined — each builder constructed its own scopes, groupings,
and aggregations directly. This made it hard to swap the underlying data
source without duplicating builder logic.
This PR extracts all raw-event querying into a `Reports::RawDataSource`
behind a `Reports::DataSource` factory. Builders now call
`data_source.timeseries`, `.aggregate`, or `.summary` instead of
constructing queries themselves. Behavior is identical —
`DataSource.for(...)` returns `RawDataSource` in all cases today.
The timeseries path had two separate builders (`CountReportBuilder`,
`AverageReportBuilder`) that were selected via a metric-name case
statement in `Conversations::BaseReportBuilder`. These are replaced by a
single `ReportBuilder` that delegates to the data source. The metric
type (count vs average) is now decided inside the data source, not the
builder.
Summary builders similarly moved their inline SQL into
`RawDataSource#summary`, which returns a unified hash keyed by dimension
ID.
the rollup read path.
## Flow
### Before
```
ReportsController ──▶ case metric ──▶ AverageReportBuilder ──▶ inline SQL ──▶ DB
└──▶ CountReportBuilder ──▶ inline SQL ──▶ DB
SummaryController ──▶ AgentSummaryBuilder ──▶ inline SQL ──▶ DB
└──▶ InboxSummaryBuilder ──▶ inline SQL ──▶ DB
└──▶ TeamSummaryBuilder ──▶ inline SQL ──▶ DB
```
### After
```
ReportsController ──▶ ReportBuilder ──┐
├──▶ DataSource.for ──▶ RawDataSource ──▶ DB
SummaryController ──▶ SummaryBuilder ──┘
```
### Expected (after rollup read path)
```
ReportsController ──▶ ReportBuilder ──┐
├──▶ DataSource.for ──▶ RawDataSource ──▶ reporting_events
SummaryController ──▶ SummaryBuilder ──┘ └──▶ RollupDataSource ──▶ reporting_events_rollups
```
### What changed
- `Reports::DataSource` factory + `Reports::RawDataSource`
- `TimezoneHelper#timezone_name_from_params` — prefers IANA name, falls
back to offset
- Unified `Timeseries::ReportBuilder` replaces `CountReportBuilder` +
`AverageReportBuilder`
- Summary builders delegate to `DataSource` instead of querying directly
### How to test
This is a pure refactor — all existing report pages (Overview, Agent,
Inbox, Label, Team) should produce identical numbers. No feature flag or
new config needed.
---------
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
Co-authored-by: Tanmay Deep Sharma <tanmaydeepsharma21@gmail.com>
Co-authored-by: Tanmay Deep Sharma <32020192+tds-1@users.noreply.github.com>
This PR adds a new standalone `GET
/api/v2/accounts/:id/reports/outgoing_messages_count` endpoint that
returns outgoing message counts grouped by agent, team, inbox, or label.
The index is already added in production.
Adds a new reporting API that returns conversation counts grouped by
channel type and first response time buckets (0-1h, 1-4h, 4-8h, 8-24h,
24h+).
- GET /api/v2/accounts/:id/reports/first_response_time_distribution
- Uses SQL aggregation to handle large datasets efficiently
- Adds composite index on reporting_events for query performance
Tested on production workload.
Request: GET
`/api/v2/accounts/1/reports/first_response_time_distribution?since=<since>&until=<until>`
Response payload:
```
{
"Channel::WebWidget": {
"0-1h": 120,
"1-4h": 85,
"4-8h": 32,
"8-24h": 12,
"24h+": 3
},
"Channel::Email": {
"0-1h": 12,
"1-4h": 28,
"4-8h": 45,
"8-24h": 35,
"24h+": 10
},
"Channel::FacebookPage": {
"0-1h": 50,
"1-4h": 30,
"4-8h": 15,
"8-24h": 8,
"24h+": 2
}
}
```
---------
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
This PR added new API endpoint GET
/api/v2/accounts/:account_id/reports/inbox_label_matrix that returns
conversation counts grouped by inbox and label in a matrix format.
Supports optional filtering by date range, inbox_ids, and label_ids.
---------
Co-authored-by: Pranav <pranav@chatwoot.com>
This API gives you how many conversations exist per channel, broken down
by status in a given time period. The max time period is capped to 6
months for now.
**Input Params:**
- **since:** Unix timestamp (seconds) - start of date range
- **until:** Unix timestamp (seconds) - end of date range
**Response Payload:**
```json
{
"Channel::Sms": {
"resolved": 85,
"snoozed": 10,
"open": 5,
"pending": 5,
"total": 100
},
"Channel::Email": {
"resolved": 72,
"snoozed": 15,
"open": 13,
"pending": 13,
"total": 100
},
"Channel::WebWidget": {
"resolved": 90,
"snoozed": 7,
"open": 3,
"pending": 3,
"total": 100
}
}
```
**Definitons:**
resolved = Number of conversations created within the selected time
period that are currently marked as resolved.
snoozed = Number of conversations created within the selected time
period that are currently marked as snoozed.
pending = Number of conversations created within the selected time
period that are currently marked as pending.
open = Number of conversations created within the selected time period
that are currently open.
total = Total number of conversations created within the selected time
period, across all statuses.
This PR collapses multiple queries fetching stats from a single table to
a single query
```sql
SELECT
user_id as user_id,
COUNT(CASE WHEN name = 'conversation_resolved' THEN 1 END) as resolved_count,
AVG(CASE WHEN name = 'conversation_resolved' THEN value END) as avg_resolution_time,
AVG(CASE WHEN name = 'first_response' THEN value END) as avg_first_response_time,
AVG(CASE WHEN name = 'reply_time' THEN value END) as avg_reply_time
FROM "reporting_events"
WHERE
"reporting_events"."account_id" = <account_id> AND
"reporting_events"."created_at" >= '2025-09-14 18:30:00' AND
"reporting_events"."created_at" < '2025-10-14 18:29:59'
GROUP BY "reporting_events"."user_id";
```
### Why this works?
Here's why this optimization is faster based on PostgreSQL internals:
- Single Table Scan vs Multiple Scans: Earlier we did 4 sequential scans
(or 4 index scans) of the same data, with the same where clause, now in
a single scan all 4 `CASE` expressions are evaluated in a single pass.
- Shared Buffer Cache Efficiency: PostgreSQL's shared buffer cache
stores recently accessed pages, with this, pages are loaded once and
re-used for all aggregation, earlier with separate queries we were
forced to re-read all from the disk each time
- Reduced planning and network overhead (4 vs 1 query)
### How is it tested
1. The specs all pass without making any changes
2. Verified the reports side by side after generating from report seeder
#### How to test
Generate seed data using the following command
```bash
ACCOUNT_ID=1 ENABLE_ACCOUNT_SEEDING=true bundle exec rake db:seed:reports_data
```
Once done download the reports, checkout to this branch and download the
reports again and compare them
There was a fundamental difference in how resolution counts were
calculated between the agent summary and timeseries reports, causing
confusion for users when the numbers didn't match.
The agent summary report counted all `conversation_resolved` events
within a time period by querying the `reporting_events` table directly.
However, the timeseries report had an additional constraint that
required the conversation to currently be in resolved status
(`conversations.status = 1`). This meant that if an agent resolved a
conversation that was later reopened, the resolution action would be
counted in the summary but not in the timeseries.
This fix aligns both reports to count resolution events rather than
conversations in resolved state. When an agent resolves a conversation,
they should receive credit for that action regardless of what happens to
the conversation afterward. The same logic now applies to bot
resolutions as well.
The change removes the `conversations: { status: :resolved }` condition
from both `scope_for_resolutions_count` and
`scope_for_bot_resolutions_count` methods in CountReportBuilder, and
updates the corresponding test expectations to reflect that all
resolution events are counted.
## About timezone
When a timezone is specified via `timezone_offset` parameter, the
reporting system:
1. Converts timestamps to the target timezone before grouping
2. Groups data by local day/week/month boundaries in that timezone, but
the primary boundaries are sent by the frontend and used as-is
3. Returns timestamps representing midnight in the target timezone
This means the same events can appear in different day buckets depending
on the timezone used. For summary reports, it works fine, since the user
only needs the total count between two timestamps and the frontend sends
the timestamps adjusted for timezone.
## Testing Locally
Run the following command, this will erase all data for that account and
put in 1000 conversations over last 3 months, parameters of this can be
tweaked in `Seeders::Reports::ReportDataSeeder`
I'd suggest updating the values to generate data over 30 days, with
10000 conversations, it will take it's sweet time to run but then the
data will be really rich, great for testing.
```
ACCOUNT_ID=2 ENABLE_ACCOUNT_SEEDING=true bundle exec rake db:seed:reports_data
```
Pro Tip: Don't run the app when the seeder is active, we manually create
the reporting events anyway. So once done just use `redis-cli FLUSHALL`
to clear all sidekiq jobs. Will be easier on the system
Use the following scripts to test it
- https://gist.github.com/scmmishra/1263a922f5efd24df8e448a816a06257
- https://gist.github.com/scmmishra/ca0b861fa0139e2cccdb72526ea844b2
- https://gist.github.com/scmmishra/5fe73d1f48f35422fd1fd142ea3498f3
- https://gist.github.com/scmmishra/3b7b1f9e2ff149007170e5c329432f45
- https://gist.github.com/scmmishra/f245fa2f44cd973e5d60aac64f979162
---------
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
Co-authored-by: Pranav <pranav@chatwoot.com>
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
This PR updates the report pages for agents, inboxes, and teams by
replacing charts with aggregate values (under a feature flag). Users can
click on any item to view more details if needed. Most users seem to
prefer aggregate values, so this change will likely stay.
The PR also includes a few fixes:
- The summary reports now use the same logic for both the front-end and
CSV exports.
- Fixed an issue where a single quote was being added to values with
hyphens in CSV files. Now, ‘n/a’ is used when no value is available.
- Fixed a bug where the average value was calculated incorrectly when
multiple accounts were present.
These changes should make reports easier to use and more consistent.
### Agents:
<img width="1438" alt="Screenshot 2025-01-26 at 10 47 18 AM"
src="https://github.com/user-attachments/assets/bf2fcebc-6207-4701-9703-5c2110b7b8a0"
/>
### Inboxes
<img width="1438" alt="Screenshot 2025-01-26 at 10 47 10 AM"
src="https://github.com/user-attachments/assets/b83e1cf2-fd14-4e8e-8dcd-9033404a9f22"
/>
### Teams:
<img width="1436" alt="Screenshot 2025-01-26 at 10 47 01 AM"
src="https://github.com/user-attachments/assets/96b1ce07-f557-42ca-8143-546a111d6458"
/>
---------
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
Co-authored-by: Shivam Mishra <scm.mymail@gmail.com>
Following https://github.com/chatwoot/chatwoot/pull/10604, this PR
introduces similar reporting features for Agents and Teams.
Updates in this PR:
- Added additional methods to the base class to avoid repetition.
- Improve reporting for Teams and Agents to include resolution count.
The Inbox Overview section is being updated to offer a more detailed
report, showing an overall view of the account grouped by inboxes. To
view detailed reports and access specific graphs for individual inboxes,
click on the inbox name to navigate to its dedicated report page.
---------
Co-authored-by: Sojan Jose <sojan@pepalo.com>
* feat: add heatmap component
* feat: add heatmap component
* feat: add dummy heatmap
* refactor: compact tiles
* feat: allow hour
* feat: wire up heatmap query
* feat: allow arbritrary number of weeks
* feat: update position of the widget
* chore: update heatmap title
* refactor: move traffic heatmap to overview
* chore: add comment for perf
* feat: add reconcile logic for heatmap fetching
Fetching the data for the last 6 days all the time is wasteful
So we fetch only the data for today and reconcile it with the data we already have
* refactor: re-org code for new utils
* feat: add translations
* feat: translate days of the week
* chore: update chatwoot utils
* feat: add markers to heatmap
* refactor: update class names
* refactor: move flatten as a separate method
* test: Heatmap Helpers
* chore: add comments
* refactor: method naming
* refactor: use heatmap-level mixin
* refactor: cleanup css
* chore: remove log
* refactor: reports.js to use object instead of separate params
* refactor: report store to use new API design
* refactor: rename HeatmapHelper -> ReportsDataHelper
* refactor: separate clampDataBetweenTimeline
* feat: add tests
* fix: group by hour
* feat: add scroll for smaller screens
* refactor: add base data to reconcile with
* fix: tests
* fix: overflow only on smaller screens
* feat: translate tooltip
* refactor: simplify reconcile
* chore: add docs
* chore: remoev heatmap from account report
* feat: let Heatmap handle loading state
* chore: Apply suggestions from code review
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
* feat: update css
* refactor: color assignment to range
* feat: add short circuit
* Update app/javascript/dashboard/routes/dashboard/settings/reports/components/Heatmap.vue
---------
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
Co-authored-by: Nithin David Thomas <1277421+nithindavid@users.noreply.github.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
This change allows the admin user to fetch conversation metrics for an account, agents, and filter conversation metrics for a specific agent.
Fixes#4305
- ReportBuilder wasn't using the specified time zone for the timestamp in the
"build" method
- The ReportController spec was calling `Time.zone.today.to_time`, but
`Date#to_time` disregards the zone from `Time.zone` and reverts to the system
time zone.
